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ABSTRACT 


This  thesis  proposes  a  hierarchy  of  documentation 
for  combat  models.  It  begins  by  examining  criticisms 
and  credibility  of  combat  models  to  establish 
underlying  causes  and  effects,  and  then  it  addresses 
model  proliferation  and  ever  increasing  complexity  as 
they  affect  one  's  ability  to  understand  and  transfer 
models.  A  methodology  for  determining  whether  or  not 
a  model  is  applicable  to  a  specific  problem  is 
presented,  as  are  examples  of  potential  problem 
areas.  Current  documecitation  practices  are  examined 
for  conditions  that  limit  the  transferability  of 
models  and  contribute  to  the  credibility  problem. 
The  aoove  examinations  have  lead  to  a  proposed 
three-tier  hierarchy  of  documentation,  including  for 
the  analyst  documentation  that  is  presented  from  the 
context  of  discovery  rather  than  from  the  traditional 
context  of  justification.  Recommendations  are  made 
for  supplemental  studies  to  examine  related  issues. 
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I.   BACKGROUND  AND  INTRODUCTION 


A.   BACKGROUND 


Motivation  for  this  thesis  was  initially  provided  by 
Professor  James  G.  Taylor,  who  wanted  to  use  an  existing 
fully  operating  large-scale  combat  model  as  a  teaching 
vehicle  in  a  com  bat- modell ing  course  in  the  Operations 
Analysis  Curriculum  at  the  Naval  Postgraduate  School  (NPS) . 
The  original  plan  was  to  acquire  the  ATLAS  (A  Tactical, 
Logistical,  and  Air  Simulation)  model  from  the  U.  S.  Army's 
Concepts  Analysis  Agency,  convert  it  for  use  on  the  NPS  IBM 
360  computer  and  develop  a  manual  for  the  setup  and  running 
of  the  model  as  part  of  the  course.  Then  the  attrition  and 
movement  routines  of  the  model  were  to  be  analyzed  as  a 
formal  thesis. 

The  project  was  especially  appealing  to  the  author 
because  it  conformed  with  a  fundamental  belief  of  his,  that 
rather  than  developing  new  models  if  an  existing  model  is 
appropriate  for  a  given  analysis  it  should  be  acquired  and 
used.  The  concept  is  not  original  but  was  based  on 
recommendations  of  the  Army  Models  Review  Committee.  [1] 
Acquisition  and  use  of  ATLAS  at  the  NPS  would  be  an 
application  of  the  recommended  concept  of  transferring  an 
existing  model  whenever  its  use  is  feasible. 

In  late  Feb.  1979,  the  ATLAS  model  arrived  via  a 
magnetic  tape,  and  the  author  proceeded  to  execute  the  above 
plan.     After  the  expenditure  of  about  four   man-months   of 


effort  by  the  author  and  computer  center  programmers  and  190 
minutes  of  CPU  time,  ATIAS  was  successfully  compiled  and 
linked.  This  expenditure  of  effort  was  greatly  in  excess 
of  the  expected  time  to  complete  the  task,  given  that  the 
model  has  been  in  existence  for  tan  or  more  years  and  is 
considered  to  be  "simple"  compared  with  other  theatre-level 
models.  Why  had  its  transfer  required  such  expenditure  of 
effort? 

In  reflecting  upon  the  above  question,  the  author 
realized  that  even  though  a  model  may  have  been  used  for 
many  years,  its  transfer  can  still  be  hampered  by  limited 
documentation.  This  thesis  will  relate  the  author's 
experience  with  ATLAS  and  his  subsequent  investigation  of 
how  documentation  limits  the  transfer  of  existing  models 
between  agencies.  It  will  also  examine  the  related 
problems  of  model  complexity,  proliferation,  credibility, 
transparency,  and  transferability.  Based  on  this 
examination  and  the  experiences  of  the  author  during  seven 
years  of  operation  research  related  assignments,  a  new 
concept  of  documentation  to  improve  the  transferability  of 
combat  models  is  proposed. 


B.   DEFINITIONS 


In  research  conducted  for  this  thesis  the  author 
discovered  that  in  modeling  the  same  word  can  have  many 
connotations.  For  purposes  of  clarity  the  author  adopted 
the  definitions  of  modeling  terras  listed  in  the  glossary  of 
Ref.  1.  This  does  not  imply  that  they  are  the  only  or  best 
definitions,    they   are   only    used    as   a    point    of   departure. 

Within  the  following  text  there  will  be  references  to 
combat      models,      theatre-level,    large-scale,    and   small-scale 


models.  For  purposes  of  the  remarks,  discussions,  and 
recommendations  contained  herein,  the  words  can  be 
considered  synonymous.  Since  the  initial  impetus  for  the 
thesis  concerned  work  with  a  theatre-level  model  this  term 
readily  crept  into  the  author's  dicussion.  However,  all 
these  forms  of  models  are  just  subsets  of  the  concept  of  a 
combat  model.  What  is  addressed  throughout  the  thesis  is 
combat  modeling.  Particular  comments  referring  to  one  of 
the  subsets  are  just  as  applicable  to  combat  models  as  a 
whole. 

Although  some  remarks  are  addressed  to  the  DOD,  others 
to  the  Department  of  the  Army,  each  is  applicable  to  the 
other  as  well  as  the  other  services.  The  discussions, 
conclusions,  and  recommendations  are  equally  applicable  to 
other  complex  models  as  well  as  models  in  general. 


C.   MODELING  AND  MODELS 


What  is  modeling?  According  to  Morris  [2]  modeling  is 
an  intuitive  process  through  which  an  analyst  arrives  at  a 
model.  On  the  other  hand,  a  model  is  an  inanimate  object, 
an  abstraction  of  reality  [3],  that  is  used  by  the  analyst 
to  answer  questions  about  some  future  state  of  a  process. 
This  differentiation  between  modeling  and  a  model  is 
necessary  to  understand  the  implications  of  the  facts 
presented  and  the  conclusions  drawn  in  this  thesis. 

To  develop  a  model,  the  modeler  goes  through  a  process 
of  discovery.  This  process  is  the  trial  and  error  procedure 
in  which  the  designer  tries  to  abstract  the  key  elements  of 
reality.  Once  he  has  done  this  and  validated  his  model,  the 
logical  reconstruction  of  events  leading  to  the  model  are 
documented  in  the  context   of   justification.   This   logical 


reconstruction  has  little  if  anything  to  do  with  the  actual 
process  followed  in  building  the  model.  No  attempt  is  made 
to  verbalize  the  actual  psychological  process,  the  problems 
encountered,  or  dead  ends  pursued. 

Over  time  the  initial  simple  model  proceeds  through  a 
process  of  enrichment  or  elaboration.  Through  this  process 
the  model  is  modified  and  moved  in  evolutionary  fashion 
toward  a  more  elaborate  representation  of  reality;  in  the 
process  the  model  becomes  more  complex  as  it  seeks  to 
reflect  the  complexity  of  the  reality  it  represents.  Each 
time  the  model  is  enriched  the  reasons  why  and  how  it  is 
enriched  should  be  documented.  For  a  discussion  of  the 
modeling  process  see  R ef .  2. 


D.   INT2ODUCTI0N 


Since  ancient  times  military  planners  have  used  wargames 
to  investigate  various  aspects  of  possible  future  military 
operations.  Historical  development  of  war  games  can  be 
found  in  Hef.  4  or  other  histories  of  war  games  in  the  open 
literature.  The  oldest  known  form  of  the  wargame  is  a 
Hindu  chess-like  game  called  "Chat uranga".  Modern  war 
games  had  their  beginning  in  1664  thru  the  games  developed 
by  Christopher  Werkmann  called  military  chess.  In  the 
twentieth  century  the  greatest  proponent  of  war  games  till 
the  end  of  WWII  was  the  German  Army.  Through  the 
development  of  the  digital  computer  during  the  war  a  new 
facet  of  wargaming  became  possible.  The  use  of  the  computer 
greatly  reduced  the  time  and  effort  required  to  conduct  a 
war  game.  Computer  assisted  or  computer  run  war  games 
provide  a  means  of  gaining  insights  and  experience  in 
military  problem  solutions.  [5]  These  computer  war  games 
help   evaluate   new   weapons   systems,   study   current    and 
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proposed  military  organizations  and  investigate  the  possible 
outcomes  of  future  conflicts  given  particular  weapons, 
organizations,  tactics,  and  enemy  forces.  The  basis  of  such 
computer  wargames  is  the  combat  model.  Although  there  are 
many  military  applications  of  combat  models,  this  thesis  is 
primarily  concerned  with  those  applicable  to  military 
strategy  and  force  planning. 

Military  strategy  and  force  planning  has  matured  since 
BWII.  Prior  to  WWII  anilitary  technology  evolved  slowly; 
those  responsible  for  strategy  and  planning  could  easily 
gain  all  the  necessary  knowledge  about  the  relationship  of 
military  forces  and  weapon  systems  to  national  security  from 
books  or  personal  experience.  Within  one  lifetime  the 
amount  of  technological  change  was  not  sufficient  to  render 
experience  invalid.  The  military  did  not  plan  on 
technological  change;  it  merely  adjusted  to  it. 

During  and  subsequent  to  WWII,  the  rate  of  change  of 
military  technology  began  to  increase  in  an  almost 
exponential  manner.  A  lifetime  of  experience  could  became 
obsolete  in  a  few  years;  now  mere  adjustment  to  change  was 
not  sufficient,  the  military  had  to  plan  for  change.  This 
revolution  in  technology  was  incubated  and  nurtured  by  the 
advent  of  the  digital  computer:  to  keep  pace  with 
technology  and  its  impact  on  strategy  and  force  planning  the 
digital  computer  was  adopted  as  a  planning  tool.  With  it 
the  means  to  assess  and  adapt  technological  change  and 
incorporate  it  into  military  strategy  and  force  planning  was 
possible .  [ 6  ] 

Modern  day  force  planning  has  become  largely  an 
analytical  process  that  necessarily  employs  the  digital 
computer.  The  digital  computer  has  been  incorporated  into 
many  of  the  myriad  aspects  of  military  planning  and  decision 
making   in   order   to   provide   a   scientific  basis  to  these 
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activities.  In  this  thesis,  the  computer  is  considered  only 
as  a  computational  aid  for  combat  models.  Tha  outputs  of 
these  models  are  used  as  aids  to  decision  makers.  Strategy 
and  force  planning  depend  primarily  on  large-scale  combat 
models,  which  due  to  theic  complexity  require  the  use  of 
digital  computers.  Models  can  be  of  various  types  or 
classes;  there  is  no  agreed  upon  system  of 
classification.  Discussions  of  classification  of  combat 
models  are  contained  in  the  literature.  [3,7  8]  This  study 
is  concerned  with  large  scale  computer  simulations  and 
analytical  models  as  defiaed  in  Ref.  3. 


S.   SUMMARY 


This  chapter  has  discussed  the  background  of  this 
thesis,  and  it  has  provided  some  basic  definitions  and  a 
general  history  of  combat  modeling  to  the 
present.  Subsequent  chapters  will  now  address  the  following 
important  aspects  of  aodels  used  for  defense  planning: 
criticism  and  credibility;  proliferation,  complexity  and 
transparency,  and  requirements  to  transfer  a  model.  Next 
a  common  element  of  each  chapter,  lack  of  adequate 
documentation,  is  identified  as  a  contributing  cause  to  each 
of  the  conditions  discussed.  This  is  followed  by  a 
proposed  concept  of  documentation  that  will  alleviate  many 
of  the  problems  discussed  in  this  thesis. 
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II.       CRITICISES    AND    CREDIBILITY    OF    COMBAT    MODELING 


A.       CAUSES    OF   CRITICISM 


Critics  and  criticism  of  defense  analysis  and  its  tools, 
of  which  modeling  is  just  one,  are  ever  present.  Criticism 
has  not  abated  since  the  early  sixties;  it  continues  to  the 
present  and  threatens  funding,  the  life  blood  of  analysis, 
in  the  Department  of  Defense.  Informed  private  citizens  and 
activitists  groups  have  ittacked  the  propriety  of  defense 
analysis  and  DOD  decisions  based  on  analysis.  The  poor 
public  image  was  cited  as  a  contributing  factor  to  the  poor 
reception      of      analysis      in      Congress.  Because   of   seeming 

inconsistencies  in  analysis.  Congress  has  become  skeptical 
and  disenchanted  and  has  questioned  the  utility  of 
analysis  .    [ 9  ] 

According  to  surveys  and  studies  conducted  early  in  this 
decade  [10]  activity  aad  expenditures  on  gaming  and 
simulation  peaked  in  the  middle  1960' s  and  were  on  a  slight 
decline  since  then.  Thesa  investigations  indicated  that 
machine  simulation  had  generally  been  oversold  and  at  that 
time  Operations  Rasearch  and  modeling  were  undergoing  a 
critical  self-examination.  [11]  Tha  criticisms  were  many 
and  encompassed  a  wide  variety  of  cause  and  effect 
relationships.  Those  of  relevance  to  this  study  revolve 
about        the        intar-related        areas        of  transferability, 

complexity,    proliferation   and   documentation. 
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As  the  use  of  combat  models  increased,  so  did  their 
initial  acceptance  and  importance,  and  in  turn  the 
complexity  of  these  models  has  also  increased.  Complexity 
has  manifested  itself  in  various  forms:  inputs,  types  of 
models,  language,  detail  of  actions  and  conditions 
simulated,  simulated  decision  making  and  computing 
machines.  Eventually  the  complexity  of  existing  models, 
poor  transparency,  and  difficulty  of  transferability  caused 
numerous  models  to  be  developed  that  modeled  the  same  or 
very  similar  combat  phenonema.  Unfortunately,  at  this 
juncture  in  the  development  of  combat  models  (late  60»s, 
early  70*s)  criticism  and  dwindling  credibility  occurred. 
This  came  about  because  for  reasons  not  easily  recognizable 
or  understandable  to  decision  makers,  models  supposedly 
modeling  the  same  combat  process  under  the  "same"  conditions 
produced  different  and  at  time  conflicting  results. 

Analysis  impLies  rigor  and  association  with  the 
scientific  method,  yet  standards  seemed  to  have  waned; 
strict  adherance  to  standards  of  scientific  rigor  and 
discipline  were  less  than  tenacious.  Often  analyses  and 
models  produced  had  methodological  flaws.  Often  these  flaws 
were  not  discovered  until  after  an  analysis  had  been 
accepted  and  decisions  made.  In  studies  involving  models, 
one  of  the  contributing  factors  to  this  situation  was  lack 
of  detailed  understanding  of  the  model.  Other  contributing 
factors  were  the  pressure  of  time  and  limited 
distribution.  In  the  rush  to  meet  deadline  the  quality  of 
the  work  was  often  sacrificed.  By  not  distributing  a  study 
or  model  to  other  agencies  the  extra  set  of  eyes  that  can 
see  a  fatal  flaw  through  an  unbiased  view  were  never  used. 

The  use  of  more  than  one  model  has  often  resulted  in 
decision  makers  being  confronted  by  seemingly  contradictory 
results   of   different   analyses   using    different    models 
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addressing  the  same  problem.  No  wonder  that  they  have  the 
impression  that  the  models  and  methods  used  by  analysts  may 
not  be  very  objective  guides.  Yet  most  analysts  will  argue 
that  a  detailed  comparison  of  models,  their  assumptions, 
inputs  and  calculations,  show  that  the  results  are  not 
really  contradictory.   Differences  in   outputs  are   usually 

m 

the  direct  result  of  differences  in  assumptions  and  methods 
of  processing  the  input  data.  This  in  turn  presents  the 
analyst  with  two  challenges:  First,  he  must  recognize  and 
understand  these  seeming  contradictions,  and  then  he  must 
resolve  and  convey  to  the  decision  maker  these 
differences.  Dr.  Wilbur  B.  Payne,  former  Deputy 
Undersecretary  of  the  Army  for  Operations  Research  speaking 
from  the  point  of  view  of  the  decision  maker  who  tries  to 
draw  valid  conclusions  from  analyses  that  use  such  models, 
has  argued  that  he  has  frequently  seen  such  apparently 
contradictory  results  coming  from  various  models  addressing 
essentially  the  same  problem.  Furthermore,  the  decision 
making  process  in  large  organizations  is  such  that  the 
detailed  comparison  ,  if  it  is  ever  done  ,  and  resolution  of 
seeming  conflicts  usually  does  not  reach  tha  decision 
maker.  Hence,  the  decision  maker  is  left  with  the 
conclusion  that  simulation  results  are  not  consistant  and 
therefore  of  dubious  reliability.  [12] 

Contradictory  results  of  combat  models  is  a  factor  cited 
by  Huber  [12]  that  has  caused  the  very  credibility  of  combat 
modeling  to  be  questioned.  Under  ideal  conditions  a  model 
should  be  directly  connected  with  a  continuing  experimental 
program  and  should  reasonably  relate  to  other  models  that 
simulate  the  same  or  similar  processes.  The  user  must  be 
especially  watchful  in  this  respect  because  the  combat 
process  does  not  easily  lend  itself  to  establishing  a 
continuing  experimental  program.  Many  combat  models  are 
neither  built  nor  used  with  any  forethought  given  to  their 
connection   to  other  models.   Each  generally  turns  out  to  be 
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a  totally  independent  data  generator.  This  precludes  any 
meaningful  experimental  feedback  in  the  on-going  prediction 
process  and  results  in  a  mass  of  unrelated  and  often 
contradictory  data  generated  by  many  models. 

The  Army  Models  Review  Committee  report  was  the  seminal 
publication  on  theatre-level  combat  models  for  the 
seventies.  From  it  spraug  a  decade  of  proposals  and  counter 
proposals  concerning  theatre-level  combat  models.  However, 
if  one  scrutenizes  the  literature  of  the  previous  decade  a 
feeling  of  deja  vu  surfaces.  Many  of  the  ideas  and 
arguments  sounded  in  the  seventies  are  in  the  literature  of 
the  sixties.  For  an  example  of  the  problems  foreseen  and 
warnings  given,  see  Ref.  13. 


3.   IMPORTANCE  OF  ASSUMPTIONS 


One  of  the  key  facets  of  any  model  is  the  assumptions 
that  go  into  its  development.  The  importance  of 
assumptions  w-hether  in  models  or  otherwise  was  recognized 
early  in  the  development  of  systems  analysis.  When  Mc 
Namara  was  Secretary  of  Defense,  a  continuing  effort  was 
made  to  insure  assumptions  incorporated  into  models  were 
both  explicit  and  consistent.  Whether  comparing  force 
structure  or  strategy,  Mc  Namara  considered  it  possible  to 
select  assumptions  that  will  make  any  proposed  weapon  system 
or  organization  look  optimal.  [6]  Likewise,  experience  has 
shown  that  there  is  no  single  "right"  set  of 
assumptions.  There  exists  an  almost  infinite  set  of 
assumptions  each  more  or  less  defensible.  What  is  important 
is  that  the  assumptions  used  in  various  submodels  of  a  model 
are  consistent.  A  model  should  not  operate  with  one  set  of 
underlying  assumptions  in  one  submodel,  while  another 
submodel  operates  with  a  fundamentally  different  set. 
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Because  there  is  no  emperical  data  concerning  modern 
large  scale  combat,  analysis  is  relied  upon  to  produce 
insights  into  the  effects  of  existing  or  proposed  weapons, 
force  structures,  and  strategies.  Using  theatre-level 
combat  models,  the  analyst  can  examine  proposed  weapons  and 
force  structures  and  possible  outcomes  can  be  forcast. 
Studies  in  the  DOD  [6]  have  shown  that  when  alternative 
force  structures  and  weapons  are  examined  using  different 
models,  it  is  difficult  to  determine  whether  differences  in 
outcomes  are  due  to  differences  in  weapons  and  organizations 
or  assumptions  and  models  used.  Confusion  can  be  reduced 
if  each  alternative  is  examined  in  a  consistent  manner  by 
using  the  same  model  and  assumptions.  Differences  that 
occur  can  then  be  attributed  to  the  basic  structure  of  the 
force  options.  The  results  can  then  be  analyzed  and 
understood  in  light  of  the  method  of  calculation  (i.e.  the 
model)  used  and  its  inherent  assumptions.  Greater  insights 
into  the  effects  of  weapon  and  force  structure  alternatives 
on  combat  outcomes  can  be  gained  by  repeating  the  experiment 
using  an  alternative  model. 

When  more  than  one  model  is  used  for  the  same  force 
structure  analysis  often  different  results  are  obtained. 
These  different  results  ace  caused  by  the  different  inherent 
assumptions  of  each  model.  If  these  assumptions  are  known 
informed  discussion  can  take  place  because  differences  can 
be  resolved  through  evaluation  of  the  assumptions.  If  the 
assumptions  are  valid  and  acceptable,  then  the  results  must 
be  accepted  as  the  logical  conseguence  of  the  assumptions. 
Assumptions  considered  to  be  unacceptable  by  decision  makers 
can  be  eliminated  by  modifying  the  model.  What  quickly 
becomes  evident  is  that  a  model  produces  a  result  based  on 
its  inherent  assumptions;  an  equally  defeasible  but 
different  set  of  assumptions  used  in  a  model  will  produce 
another  result,  which  may  or  may  not  be  the  same. 
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The  DOD  is  quite  awars  of  this  aspect  of  modeling  and 
has  emphasized  that  there  is  more  than  one  set  of 
assumptions  that  can  be  used  with  a  given  model.  It 
requires  that  all  parties  to  the  decision  making  process  be 
aware  of  all  assumptions  leading  to  a  result.  To  achieve 
this  goal  the  DOD  has  required  that  all  assumptions  be 
explicit,  reasonable,  and  consistent.  This  requires 
ferreting  out  hidden  assumptions  in  models  and  insuring  that 
the  model  indeed  models  what  it  claims  to  model.  For 
interesting  examples  of  the  importance  of  assumptions  see 
Ref.  6. 


C.   MODEL  EVALUATION 


Many  solutions  to  the  problem  of  model  evaluation  have 
been  recommended;  one  concept  suggests  the  use  of  full  time 
dedicated  independent  reviewers  as  a  way  of  improving  the 
mechanics  of  quality  control.  [1]  A  reviewer  provides  a 
means  through  which  the  analyst's  model  is  scrutinized;  the 
assumptions  and  methodology  are  checked  for  internal 
consistency,  unwarranted  inferences,  and  clarity  of 
presentation  so  that  a  determination  can  be  made  whether  or 
not  the  model  is  a  plausible  representation  of  the  real 
world.  If  it  is  a  large  complex  model,  the  methodology 
should  be  clear,  even  to  the  point  of  sample  calculations  to 
guide  the  reviewer  and  analyst  through  the  algorithms. 
Equally  important  is  that  input  data  and  assumptions  be 
explicit.  If  unnecessary  proliferation  is  to  be  avoided, 
existing  models  must  be  made  understandable  to  potential 
users  and  evaluated  in  some  manner.  It  should  not  be 
necessary  to  create  a  new  model  just  because  an  existing 
model  could  not  be  transferred  oc  understood  due  to 
inedquate  documentation. 
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In  his  study,  Gass  [  1  4 ]  proposes  an  elaborate  approach 
to  evaluation  of  complex  models.  Therein  he  highlights  the 
need  for  model  implementation  and  maintenance  procedures  as 
well  as  documentation  of  the  model  and  the  total  modeling 
process.  Suggested  documentation  of  the  process  includes 
describing  model  objectives,  assumptions,  results,  data 
sources,  recommendations,  etc.  With  such  documentation  the 
model  can  hopefully  be  evaluated  and  analysts  can  determine 
whether  or  not  the  modal  is  valid  for  the  problem  at 
hand.  However,  Gass  found  that  for  lost  complex  computer 
models,  organizational  exigencies  and  real-world  pressures 
do  not  enable  modelers  to  develop  the  necessary 
documentation. 

Gass  has  stressed  the  need  to  validate  models  at  three 
distinct  levels,  technical,  operational,  and 
dynamic.  Technical  validity  is  an  assessment  of  model, 
data,  logical,  mathematical  and  predictive 
validity.  Operational  validity  is  an  assessment  of  errors 
and  divergences  found  under  technical  validity  and  the 
robustness  of  the  model  (i.e.  whether  or  not  the  model  can 
produce  bad  answers  for  proper  ranges  of  parameter  values) . 
Dynamic  validity  is  the  method  oy  which  the  model  will  be 
maintained  during  its  life  cycle  so  that  it  continues  to  be 
an  acceptable  representation  of  the  real  system.  It 
includes  the  process  through  which  the  model  structure  is 
changed  and  validated.  The  ability  to  accomplish  such 
validation  will  facilitate  and  enhance  the  use  of  models  by 
anaylsts  and  decision  makers  other  than  those  directly 
responsible  for  the  development  of  the  model.  Fundamental 
to  this  process  is  detailed  understanding  of  the  model. 
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D.   SUMMARY 


If  a  model  is  to  be  of  value  it,  mast  be  accepted  by  the 
decision  maker.  It  is  incumbent  upon  the  analyst  to  provide 
the  means  of  acceptance.  Sufficient  documentation  must  be 
provided  by  the  model  designer  to  enable  the  analyst  to 
understand  its  methodology  and  structure.  Key  to 
credibility  is  objective  evaluation.  Documentation  must 
provide  insights  into  the  assumptions  and  functioning  of  a 
model,  so  that  it  can  be  evaluated.  Understanding  and 
evaluation  are  complicated  by  complexity.  Complexity  and 
insufficient  documentation  can  cause  analysts  to  design  new 
models  rather  than  use  an  existing  model.  The  existence  of 
inadequately  documented  models  describing  the  same  reality 
is  the  basic  cause  of  the  criticism  and  lack  of  credibility 
of  combat  models.  The  next  chapter  will  discuss  complexity 
and  how  and  why  it  contributes  to  unnecessary  proliferation. 
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III.   PROLIFERATIDli  COMPLEXITY^.  ^ND  TRANSPARENCY 


Unnecessay  proliferation  is  th  =  cause  of  some  of  the 
criticisms  of  combat  modeling.  Unnecessary  proliferation 
means  that  a  new  model  is  created  because  inadequate 
documentation  precludes  the  use  of  an  existing  model  which 
otherwise  would  be  adequate  for  the  task  at 
hand.  Inadequate  documentation  either  prevents 
understanding  of  the  methodology  and  structure  of  the  model 
or  prevents  the  cost-effective  transfer  and  conversion  of 
the  model  from  one  agency  and  computing  system  to 
another.  The  latter  condition  is  the  one  encountered  by  the 
author  during  the  conversion  of  ATLAS.  Better  documentation 
would  have  significantly  reduced  the  resources  expended  on 
the  project.  Futhermore,  in  a  non-academic  environment  the 
demands  of  proceeding  with  an  impending  study  would  have 
encouraged  analysts  to  develop  a  new  model  rather  than 
struggle  with  poor  or  non-existing  documentation. 

Development  of  a  new  model  can  cause  analysts  to  remodel 
a  combat  process  using  techniques  and  methodology  that 
already  exists;  consequently  funds  are  expended  without 
advancing  modeling.  Irrespective  of  any  advancements  in 
techniques  or  improvements  in  methodology  developed  in 
remodeling  an  existing  process,  a  major  portion  of  the 
effort  is  devoted  to  redoing  the  basics.  Time  and  money 
spent  in  redoing  the  basics  are  lost  as  far  as  improving 
models  and  modeling  is  ooncerned.  In  the  DOD  (especially 
the  Department  of  the  Army)  there  has  been  and  continues  to 
be  a  shortage  of  trained  analysts.  An  obvious  approach  to 
alleviate  this  problem  would  be  to  eliminate  any  projects  to 
develop   models   that   would  be  redundant  and  share  existing 
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models.   However,  the  ability  to  share  a   model   is   greatly 
influenced  by  its  complexity. 

When  examining  the  practicality  and  feasibility  of  model 
sharing  most  emphasis  has  been  placed  on 
documentation.  Huber  [8]  lists  poor  documentation  and  high 
personnel  turnover  as  a  prime  reasons  why  models  are  not 
exchanged.  The  criterion  used  to  determine  whether  or  not  a 
model  is  transferable  is  documentation  sufficient  to  allow 
the  recipient  organization  to  be  able  to  run  the  model 
without  an  inordinate  amount  of  decoding  and 
deciphering.  Without  ade:juate  documentation  the  model  is  a 
puzzle  to  the  recipient.  Shubick  and  Brewer  [10]  found  that 
few  models  exist  in  a  sufficiently  documented  form  that 
would  satisfy  commercial  firm  standards  prior  to 
distribution  to  their  clientle. 


A.   PROLIFERATION  AND  HOW  TO  REDUCE  IT 


An  area  examined  by  the  Comptroller  General  in  1974, 
[15]  was  sharing  of  computerized  models.  Models  developed 
for  specific  purposes  by  one  agency  can  often  be  used  by 
other  agencies  for  simiLar  purposes.  Applicability  of 
models  to  new  situations  depends  on  their  accuracy,  purpose, 
validity,  availability  of  sufficient  documentation,  and 
capability  of  the  using  analyst.  The  1974  study  surveyed 
242  models  that  had  a  combined  cost  of  over  thirteen  million 
dollars.  An  attempt  wis  then  made  to  obtain  documentation 
for  about  one  hundred  randomly  selected  models  deemed  to 
have  use  at  more  than  the  originating  agency.  Information 
explaining  the  purpose,  mathematical  formulation,  and 
operating  instructions  were  not  available  for  approximately 
one-third  of  the  models.  The  survey  identified  the 
primary  complaints  of  model  users  (programmers  and  analysts) 
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as:  operating  instructions  not  available  or  oot  clear, 
hence,  compounding  the  already  difficult  problem  of 
preparing  a  model  for  use  on  a  different  computer; 
mathematics  of  the  model  not  clearly  explained,  hence, 
restricting  the  understanding  of  the  logic,  capabilities  and 
limitations  of  the  model;  sample  inputs  and  outputs 
nonmatching  or  nonexistent  or  do  not  correspond  with  the 
sample  data  in  documentation,  hence,  verification  and 
validity  of  the  model  difficult  to  determine;  flow  chart 
explaining  logic  not  provided  or  not  current,  henze,  complex 
subroutines  not  easily  understood.  The  investigation 
determined  that  benefits  can  be  obtained  by  sharing  computer 
models,  however,  before  models  can  oe  shared  adequate 
documentation  must  be  prepared.  Such  documentation  enabled 
the  acquiring  agency  to  determine  whether  a  model  met  its 
needs  and  was  the  primary  factor  to  successful  conversion 
and  operation  of  the  model  on  a  different  computer.  The 
potential  for  a  cost-effective  transfer  is  severely  limited 
in  the  absence  of  adequate  documentation.  When  such 
documentation  is  available  to  potential  users  of  an  existing 
model  great  savings  are  realized.  An  exampla  was  the 
transfer  of  a  complex  communications  traffic  analysis  model 
from  an  Air  Force  agency  on  the  West  Coast  to  the  Systems 
Development  Command  at  Hanscom  Air  Field  in  Bedford, 
Massachusetts.  [15] 

Joint  usage  of  existing  models  not  only  increases  the 
availability  of  trained  individuals  to  do  further  research 
and  analysis,  but  it  reduces  the  opportunity  that  different 
factions  of  the  same  organization  are  working  at  cross 
purposes.  Conflicting  concepts  and  proposals  are  necessary 
to  vitalize  an  organization  and  make  it  viable,  but 
developing  a  conflicting  position  that  could  be  resolved 
prior  to  the  expenditure  of  great  sums  of  money  and 
analytical  talent  is  a  waste.  The  sharing  of  a  model  or 
models  between   two   conflicting   agencies   allows   each   to 
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understand  the  underlying  basis  for  their  respective 
positions  concerning  an  analysis  Df  a  decision  making 
situation.  While  sharing  may  not  resolve  the  conflict  it 
certainly  will  preclude  the  expenditure  of  funds  in  the 
independent  acquisition  of  information  that  is  already 
available  through  the  medium  of  an  existing  model.  Sharing 
has  eliminated  retracing  steps  already  taken  and  dead  ends 
already  discovered  when  applied  in  other  fields  ,  applied  to 
combat  modeling  sharing  will  provide  these  minimal  returns 
and  has  the  prospective  of  providing  even  greater 
returns.  If  the  basics  are  not  reprocessed  then  more  time 
and  money  is  available  fDr  modifications  to  enhance  the 
capability  of  an  existing  model,  correct  known  deficiencies, 
or  identify  suspected  deficiencies.  Sharing  has  promise  to 
improve    the    economies    of   modeling.    [16] 

Before   sharing    can    be   achieved   certain   basic    conditiions 
must      prevail.  Five      necessary      conditions   [17]    to    model 

sharing    have   been    found.      They      are; 

(1)  a      computer      able      to      use   the    program    with    minimal 
modification ; 

(2)  an   adequate    facility    to    run   the    model; 

(3)  adequate    documentation   of   the    original   model; 

(4)  sufficient    analysts    with   technical   competence; 

(5)  formalized     arrangements      for      sharing      costs        and 
responsibility   for  costs   and   coordination. 
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B.   THE  EFFECTS  OF  COMPL2XITY  ON  TRANSPARENCY 


Complexity  is  a  diohotomous  issue  in  itself.  Gass 
perceives  an  increasing  use  of  more  complex  models  [14], 
while  Hardison,  the  Deputy  Under  Secretary  of  the  Army  for 
Operations  Research  at  the  15th  Annual  US  Army  Operation 
Research  Symposium  called  for  less  complexity. 
Disatisf action  of  both  senior  military  and  civilian  decision 
makers  with  complex  models  and  studies  was  emphasized  by 
Hardison.  [18]  These  decision  makers  are  convinced  that 
Army  models  and  the  studies  they  support  are  too  complex, 
elaborate  the  obvious,  belabor  needless  details  and  overlook 
key  issues.  Timeliness  is  also  affected  by 
complexity.  Failure  to  delimit  results  in  failure  to  meet 
schedules  which  causes  something  to  be  sacrificed;  often,  in 
the  case  of  models  that  something  is  adequate 
documentation. 

A  corollary  of  the  complexity  issue  is  that  of 
transparency.  If  all  the  interactions  of  a  model  are  to  be 
understood  by  both  the  analyst  and  the  decision  maker  it 
must  be  structured  and  programmed  so  that  its  methodology  is 
easily  understood.  A  model  that  fulfills  this  requirement 
is  said  to  be  transparent.  At  the  35th  Session  of  MORS  a 
leading  cause  of  the  general  disenchantment  with 
theatre- level  models  in  recent  years  was  attributed  to  a 
lack  of  transparency  in  most  models.  The  proposed 
resolution  to  this  problem  was  to  include  in  the  model  only 
those  interactions  and  factors  that  can  be  shown  to 
influence  the  outcome.  This  in  combination  with 
mathematical  formulation  that  is  as  simple  as  possible 
should  produce  the  desired  transparency.  [19]  Yet,  at  this 
same   MORS   session   A.H.   Cordesman   OASD  (Intelligence)  in 
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discussing  theatre- level  models  remarked  that  models 
currently  being  developed  go  into  unnecessary  levels  of 
detail  in  ways  which  seriously  limit  their  value.  This  is 
partially  caused  by  intermediate  managers  and  decision 
makers  requesting  particular  attributes  be  modeled.  At 
times  modeling  efforts  deviate  from  the  maxim  that  only 
"essential"  variables  be  modeled.  As  Morris  [2]  suggests, 
the  purpose  of  a  model  is  to  include  only  thosa  variables 
that  characterize  the  process  being  modeled.  At  present 
diametrically  opposed  forces  exist;  while  expounding  the 
need  to  maintain  models  at  a  simple  transparent  level 
current  modeling  efforts  go  into  details  that  detract  from 
transparency. 

A  simple  solution  to  the  transparency-complexity  problem 
may  not  be  easily  obtained.  In  spite  of  the  professional 
rhetoric  to  the  contrary,  Gass  [14]  has  found  an  increase  in 
the  use  of  complex  models  at  all  levels  of  government  and 
industry.  He  attributes  this  to  better  trained  analysts  and 
the  development  and  refinement  of  methodologies.  Although 
simple  models  with  readily  understood  assumptions, 
relationships  and  structure  are  preferred,  Gass  contends 
that  decision  making  problem  environments  representative  of 
the  Federal  Government  sphere  cannot  be  realistically  or 
logically  contained  by  simple  models.  Furthermore,  senior 
decision  makers  generally  do  not  possess  a  detailed 
understanding  and  appreciation  of  the  methodologies  employed 
in  the  various  aodels  employed  to  assist  them  in  the 
decision  making  process.  What  is  needed  is  a  method 
through  which  the  use  and  interpretation  of  the  outputs  of 
dels  by  senior  decision  makers  is  facilitated.  A  model  is 
able  only  if  it  is  understood  and  plausible  to  analysts 
and  decision  makers.  They  (particularly  the  decision  maker) 
must  be  given  the  opportunity  to  explore  the  use  of  the 
model,  become  familiar  with  its  predictions,  and  examine  the 
relationships  and   assumptions   implied   by   the   model.   In 
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actuality  the  demands  dq  their  time  generally  preclude 
decision  makers  being  involved  at  the  detail  levels  implied 
above.  Therefore,  it  is  incumbent  upon  analysts  to  evaluate 
the  models  they  use  so  that  they  are  in  a  position  to 
recommend  to  the  decision  maker  to  use  or  not  use  the 
outputs  of  a  particular  model.  This  implies  intimate 
knowledge  of  the  essence  of  a  model. 


C.   SUMMARY 


To  reduce  unnecessary  proliferation  and  reduce  costs  the 
Comptroller  General  has  recommended  model  sharing  when 
feasible.   Sharing   a  model  requires: 

■  ability  to  use  it  with  minimal  conversion; 

■  adequate  facilities  to  run  the  model; 

■  sufficient  competent  analysts  and  programmers; 

■  adequate  documentat on; 

■  formalized   arrangements   for   cost   sharing  and 
coordination. 

Findings  indicated  that  the  great  complexity  of 
theatre- level  models  coupled  with  rapid  turnover  of 
personnel  has  resulted  in  models  being  used  as  "black  boxes" 
with  neither  the  computer  technicians  that  run  the  model  nor 
the  analysts  knowing  explicitly  what  or  how  the  model 
operated  on  the  input  data  to  provide  the  final  results  or 
output.  Hence,  the  analyst  was  unable  to  adequately  explain 
the  results  to  the  decision  maker;  with  each  occurence  the 
credibility  of  modeling  diminished.  Concurrently  models 
had  proliferated  to  such  a  degree  that  the  turnover  of 
personnel  exacerbated  an  already  critical  personnel  shortage 
situation.  Amelioration  requires  reduction  of  the  number  of 
models  in  use  and  detailed  justification  before  developing  a 
new  model.  [1}  Reduction  or  the  minimization  of  the   growth 
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of  the  model  inventory  implicitly  requires  that  models  be 
easily  and  quickly  moveable  from  one  using  agency  to 
another,  i.e.  posses  transferability.  Key  to  the  resolution 
of  the  problems  of  complexity  and  transparency  is 
documentation. 

« 

Complexity  may  be  an  unavoidable  recourse  of  combat 
modeling  because  of  the  demands  of  managers  and  decision 
makers.  Unless  complex  models  are  sufficiently  documented 
to  make  them  readily  understandable  and  usable,  analysts 
will  create  a  new  model.  Rather  than  creating  new  models 
an  atmosphere  conducive  to  the  sharing  of  models  should  be 
incouraged.  Requirements  to  transfer  a  model  are  discussed 
in  the  next  chapter. 


28 


IV.   REQUIREMENTS  TO  TRANSFER  A  MODEL 


The  Review  of  Selected  Army  Models  report  [  1 ]  proposes 
that  the  number  of  models  retained  by  the  Dept.  of  the  Army 
be  reduced  and  that  existing  models  be  used  where  possible 
before  a  new  model  is  commissioned.  To  properly  evaluate  an 
existing  model  a  method  for  analyzing  and  verifying  a 
candidate  or  candidates  from  existing  model  resources  must 
be  established.  The  more  complex  a.  model  is  the  more 
difficult  is  this  analysis  and  verification.  [14] 

Use  of  an  existing  model  requires  understanding  of 
potential  problems  due  to  model  design  as  well  those 
problems  expected  to  be  encountered  during  transfer  and 
execution.  Design  problems  can  include  lack  of  adequate 
sub-models,  failure  to  consider  key  variables,  inaccuracies 
or  lack  of  validation,  computational  difficulties,  and 
inconsistent  hidden  assumptions.  Problems  during  exection 
can  be  those  of  irrelevance,  inadequacy  of  output, 
inappropriateness  of  assumptions,  lack  of  connection  to 
other  models  and  results,  statistical  and  extrapolation 
difficulties. 


A.   METHODOLOGY  FOR  ANALYSIS  OF  EXISTING  MODELS 


3efore  an  existing  model  can  be  used  it  must  be  analyzed 
by  the  potential  user.  Examination  of  the  literature  for  a 
methodology  for  conduct  of  an  analysis  generally  produces  a 
concensus.  Such  methodologies  center  on  five  general  areas. 
[20  ]   These  are: 
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(1)  inputs  and  outputs,  i.e.  the  global  structure  of  the 
model; 

(2)  the  basic  causil  relationships  assumed  between 
variables;  i.e.  the  local  structure  of  the  model; 

(3)  the  detailed  logic  of  the  model; 

(4)  the  numerical  values  of  the  data,  and 

(5)  the  time  and  resources  required  to  exercise  the 
model. 

Additionally,  an  analysis  should  consider  any 
experimental  studies  that  allow  comparison  of  the  model 
predictions  with  the  real  world,  other  models  or  with  the 
intuitive  beliefs  of  the  decision  maker  that  will  ultimately 
be  presented  the  outputs  and  their  interpretation.  Such 
previous  studies  are  useful  in  evaluation  of  a  model  for 
application  to  new  problems  or  situations.  Unfortunately, 
so  far  as  combat  models  ace  concerned,  comparisons  with  real 
world  results  are  extremely  limited.  When  such  data  is 
available  (e.g.  WWII  and  Korea)  it  is  sparse,  subject  to 
conflicting  interpretation,  and  of  questionable  accuracy. 
[21] 

A  detailed  examination  of  the  global  structure  of  a 
model  can  forthrightly  answer  the  basic  question  of  such  an 
analysis.  Is  this  model  capable  of  examining  the  problem 
at  hand?  The  potential  user  is  interested  with  whether  the 
outputs  measure  the  desired  quantities  and  or  qualities,  and 
whether  the  desired  inputs  can  be  entered  into  the  model  in 
the  form  in  which  they  are  available.  Using  an  existing 
model  is  not  cost-effective  if  available  input  data  must 
undergo  a  costly  and  time  consuming  conversion.  There  must 
be  provisions  in  the  model  structure  to  allow  changes  in  the 


30 


input  data  to  reflect  chinges  in  the   combat   process   under 
examination. 

The  local  structure  is  examined  to  expose  important 
causal  influences  which  may  have  been  omitted  for  ease  of 
computational  effort  or  other  reasons.  This  step  is  closely 
related  to  the  abstraction  process  in  the  design  of  tne 
model.  It  is  most  dependent  on  the  detail  and  completeness 
of  available  documentation.  It  is  critical  because  the 
importance  of  a  causal  connection  can  be  quite  subtle  but 
very  pervasive. 

Examination  of  the  detailed  logic  of  a  model  identifies 
the  hypothesis  upon  which  the  model  is  based.  It  reveals 
extent  to  which  it  is  based  on  historical  and  experimental 
field  data.  It  is  another  indicator  of  the  appropriateness 
of  the  model  to  solve  the  problem  at  hand.  This  step  also 
reveals  the  presence  of  inconsistent  assumptions  or 
inappropriate  assumptions. 

Finally,  analysis  of  the  time  and  resources  required  by 
the  model  provides  the  means  through  which  the  costs 
associated  with  the  use  of  the  model  can  be  determined.  To 
make  rapid  and  accurate  astimates  the  description  of  the 
model  must  provide  explicit  information  of  time  requirements 
to  gather  and  prepare  required  inputs  as  well  as  the  time  to 
execute  the  model  given  the  inputs. 

If  models  are  to  be  truly  transferable  between  agencies 
the  above  analysis  must  be  comleted  prior  to  any  attempt  to 
transfer  a  model.  Accurate  and  complete  document*  tion  must 
be  available  if  this  is  to  be  conducted.  Such 
documentation  will  be  available  only  if  it  is  prepared 
concurrently  with  the  development  of  a  new  model  and  in 
concert  with  any  modifications  made  during  the  life  of  tne 
model. 
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B.       POTENTIAL    PROBLEMS    IN    EXISTING    MODELS 


To  adequately  analyze  an  existing  model  tha  analyst  must 
be  familiar  with  potential  problem  areas.  Parrell  [20]  has 
suggested  that   the   analyst    consider: 

(1)  the  adequacy  of  submodels,  i.e.  do  they  model  what 
they   purport    to    model; 

(2)  whether  key  variables  were  overlooked  during  the 
process    of   abstraction; 

(3)  the    possibility  Df    inherent    inaccuracies; 

(4)  possible   computational   i  ifficulties; 

(5)  whether  the  modal  has  inconsistent  and  inappr oriate 
assumptions; 

(6)  possible    inadequacy    of   output. 

Among  the  problems  of  design  can  be  found  cases  where 
the  general  model  structure  is  adequate  but  reasonable 
submodels  are  not  available.  This  is  particularly  true  of 
sub-models  involving  simulated  tactical  or  strategic 
decision  making.  Farrell  [20]  has  indicated  that  most 
diagrams  and  flow  charts  of  such  sab-models  do  not  reveal 
the   sub-model   inadequacy. 

In  designing  a  model  the  first  and  most  difficult  step 
is  insightful  abstraction  by  the  modeler  or  moielers.  To 
abstract  the  real  worli  into  a  representative  model,  key 
variables,    their    basic   causal   relations      and      interpretation 
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must  be  incorporated  into  the  model.  This  step  is  very 
difficult  to  verbalize  since  any  description  of  the  process 
to  be  modeled  will  necessarily  incorporate  the  results  of 
abstraction  in  the  elements  of  the  description.  This  step 
is  imaginative,  creative,  and  complex;  it  is  imperative  that 
it  be  documented  at  the  time  of  abstraction.  Once  a 
process  has  been  modeled  and  time  passes  the  explicit  steps 
and  reasoning  thru  which  the  abstraction  was  made  cannot  be 
adequately  described  by  the  modelers.  Failure  to  consider 
key  variables  occurs  at  this  almost  invisible  step  of 
abstracting  a  process  of  the  real-world  into  a  model  or 
sub-model. 

In  the  review  of  an  existing  model  it  is  these  almost 
invisible  steps  of  abstraction  that  must  be  thoroughly 
examined  to  insure  all  key  variables  are  included  in  the 
model.  Key  variables  can  also  be  excluded  because  of  lack 
of  adequate  sub-models  or  computational  problems  their 
inclusion  or  manipulation  would  precipetate.  The  reviewer 
or  user  of  an  existing  model  must  carefully  search  for  key 
variables  that  have  not  been  modeled  or  are  simply  thruput 
in  the  model. 

Another  pitfall  hidden  in  the  design  of  existing  models 
is  inaccuracy  or  lack  of  validation.  Often  because  of 
computational  difficulties  known  experimental  results  have 
not  been  included  in  a  particular  model.  This  occurs  most 
often  as  a  result  of  exigencies  on  the  modeler  or  modelers 
at  the  time  the  model  is  created.  Exclusion  of  such 
experimental  data  results  in  inaccuracies.  Lack  of 
experimental  support  for  portions  of  military  models  is 
another  common  cause  of  inacuracies  which  cannot  always  be 
avoided.  Likewise,  inadequate  debugging  of  the  model  can 
be  a  reality.  This  is  not  serious  as  long  as  these 
inaccuracies  are  known.  Unfortunately,  these  facts  can 
escape  a  potential  user  if  a  careful   review   of   the   model 
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design  and  its  use  is  not  made. 

A  thorough  review  of  model  documentation  can  reveal 
computational  difficulties  if  the  documentation  is 
adequate.  These  difficulties  usually  involve  either 
algorithmic  imprecision  or  excessive  date  storage 
requirements.  These  limitations  should  oe  revealed  by  the 
documentation  to  the  potential  user  so  that  proper  use  of 
the  model  can  be  made,  unexpected  or  excessive  costs  are  not 
incurred  or  another  model  can  be  selected. 

The  bane  of  analyst  using  an  existing  model  is 
inconsistant  hidden  assuaptions .  Such  assumptions  included 
as  part  of  the  overall  model  can  be  at  odds  with  those  of 
sub-models,  the  data  base,  and  data  generating  routines. 

The  most  pervasive  error  awaiting  the  unwary  user  of  an 
existing  model  is  the  inappropriateness  of  inherent 
assumptions.  This  potential  error  is  the  most  difficult 
to  detect  when  using  a  pre-designed  combat  model,  since  the 
user  is  generally  not  well  versed  in  the  process  through 
which  the  abstraction  of  the  real  world  was  made.  During 
the  abstraction  fundamental  assumptions  concerning  the 
nature  of  the  combat  process  as  well  as  assumptions  for 
computational  reasons  were  made.  Only  through  careful  study 
of  the  line  of  reasoning  followed  at  the  creation  can  a 
would-be  user  become  familiar,  with  these  assumptions.  Care 
must  be  taken  so  that  not  only  the  explicit  but  also  the 
implicit  assumptions  are  understood  and  their  effect  on  the 
combat  process  being  modeled  is 
comprehended.  Unfortunately,  the  reasoning  and  logic 
followed  in  creation  of  a  model  are  not  included  in  current 
documentation. 
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If  there  are  potential  problems  awaiting  the  unwary 
analyst  in  the  design  of  an  existing  model,  these  are  just 
the  forebodings  of  greater  problems  associated  with  the 
actual  exercise  of  the  model.  Foremost  of  these  is 
irrelevance,  that  is  caused  by  either  attempting  to  use  a 
model  on  a  problem  for  which  it  was  not  designed  or  a 
failure  to  understand  the  problem  thoroughly  enough  to 
select  an  appropriate  model. 

Concomitant  with  irrelevance  is  inadequacy  of 
output.  This  results  when  useful  information  is  inherent  in 
the  model  itself  but  actual  calculation  and  or  display  does 
not  exist.  Causes  of  such  conditions  are  selection  of  an 
improper  model  or  lack  of  full  understanding  of  the 
intricacies  of  the  model.  Proper  and  adequate 
documentation  and  careful  perusal  will  ameliorate  the 
situation. 

In  complex  simulations,  both  situations  and  key 
parameters  will  be  variei  and  the  results  examined.  The 
number  of  unique  situations  examinsd  is  limited  by  resource 
and  time  constraints,  because  of  this  the  potential  user 
must  be  sure  that  required  outputs  are  readily  available. 

Adequate  descriptions  of  the  output  of  any  random 
process  are  difficult  to  achieve  under  the  best 
conditions.  Under  time  constraints  descriptions  of  the  type 
of  output  provided  by  a  model  as  well  as  how  the  outputs  are 
collected  is  critical.  Without  it,  the  user  cannot 
correctly  interpret  the  results  obtained.  When  available 
time  and  resources  preclude  simulating  all  situations  the 
problem  of  extrapolating  or  interpolating  between  the 
particular  situations  modeled  arises.  There  is  no  adequate 
general  method  for  surmounting  this  problem;  the  problem  is 
less  severe  the  less  complex  is  the  model   being   used.   Any 
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statistical  difficulties  encountered  only  compound  the 
problem.  Surmounting  this  problem  is  a  function  of  the 
ingenuity  of  the  analyst  and  the  detail  of  the  documentation 
available  to  him.  In  the  case  of  theatre- level  combat 
models  obviously,  neither  all  situations  can  be  simulated 
nor  are  there  adequate  historical  or  experimental  data  with 
which  to  compare  the  results  obtained.  k  thorough 
accurate  and  consistent  iaterpretation  of  the  outputs  of 
such  a  model  is  highly  dependent  on  the  analysts  intimate 
familiarity  with  the  mathematical  structure  within  the  model 
as  well  as  interactions  between  the  various  input  and  output 
data.  Such  intimacy  is  obtainable  only  through  available 
documentation  if  the  analyst  uses  an  existing  model  designed 
by  someone  else. 


C.   SUMMARY 


Transfer  of  existing  models  between  agencies  is  one  way 
of  reducing  proliferation.  Prior  to  such  a  transfer  a 
potentionally  usable  model  must  be  analyzed  for 
applicability  and  approriateness .  Such  an  analysis  should 
consider : 

■  inputs  and  outputs; 

■  basic  causal  relationships; 

■  model  logic  and  structure; 

■  available  data  and  required  data; 

■  time  and  resources  required. 

Many  potential  problems  are  contained  in  an  existing 
model,  among  these  are: 

■  submodel  inadequacy; 

■  key  variables  excluded; 

■  inherent  inaccuracies; 

■  computational  difficulties; 
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■  inconsistent  assumptions; 

■  inappropriate  assumptions; 

■  inadequate  output. 

The  ability  and  to  what  degree  model  analysis  and 
consideration  of  potential  problems  can  be  accomplished  is 
determined  to  a  great  degree  by  available 
documentation.  Adequacy  of  documentation  is  considered  in 
the  next  chapter. 
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V.   STATUS  AND  ADEQUACY  OF  CURRENT  DOCUMENTATION 


A.   DEFINITIONS  AND  3ACKGR0UND 


Program  documentation  is  a  collection  of  information  to 
explain  the  design,  development,  and  maintenance  of  the 
program  as  well  as  purposes,  methods,  logic,  relationships, 
capabilities  and  limitations.  [5]  Except  for  ths  simplest 
programs  it  is  difficult,  if  not  impossible,  for  someone 
other  than  the  originator  to  determine  what  is  supposed  to 
be  accomplished  by  just  reading  the  program  code. 

Documentation  is  necessary  for:  planning,  programming, 
managing,  operating  and  evaluating  models.  It  is  absolutely 
essential  for:  guick  and  effective  changes;  use  of  the 
model  by  programmers  and  analysts  other  than  the 
originators;  understanding  of  what  is  being  done; 
interagency  program  sharing;  verification  of  proper  model 
operation.  Through  adeguate  documentation  secondary  users 
gain  an  understanding  of  a  model  and  thus  the  model  and  its 
outputs  are  rewarded  a  ley  el  of  credibility.  It  is  vital 
if  secondary  users  are  to  be  able  to  run  the  model  and  make 
necessary  modifications  of  the  program.  This  restrains 
proliferation  and  duplication  which  can  result  in  major 
savings;  besides  it  tempers  an  already  complex 
environment.  tJnf ortunately,  current  documentation  practices 
are  such  that  the  documentation  to  facilitate  the  use  of 
existing  models  generally  does  not  exist. 
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B.   CURRENT  DOCUMENT  ATIOS  PRACTICES 


Although  document ation  is  an  aspect  of  computer 
programming  that  was  recognized  from  the  inception  of 
automatic  data  processing  as  being  critical  to  successful 
programming,  it  has  been  and  still  is  a  major  problem 
area.  Unfortunately,  for  various  reasons  including  time  and 
fiscal  constraints  as  well  as  lack  of  definitive  guidance 
concerning  requirements  and  standards,  the  bulk  of  the  work 
effort  concerning  documentation  has  consisted  of  unfulfilled 
requirements.  Notwithstanding  the  fact  that  programming 
has  existed  since  the  inception  of  ENIAC  in  1944,  the  lack 
of  adequate  documentation  received  major  attention  in 
studies  concerning  models  and  simulations  as  well  as 
investigations  by  the  Comptroller  General  of  the  Army  in  tne 
late  sixties  and  early  seventies.  [10,15] 

Irrespective  of  the  aforementioned  studies  the  problem 
of  inadequate  documentation  persisted  and  was  the  subject  of 
another  investigation  oy  the  Comptroller  General  in 
1974.  [15]  Over  seventy  federal  installations  in  the 
continental  United  States,  Europe,  and  Asia  were 
surveyed.  These  included  selected  DOD  Agencies  as  well  as 
those  of  each  of  the  armed  forces. 

The  study  cited  several  problem  areas  attributable  to 
inadequate  documentation.  Increased  cost  of  operations  was 
high  on  the  list.  Beciuse  of  inadequate  documentation  use 
of  operating  programs  is  hindered  since  current  operators  do 
not  fully  understand  how  and  what  is  being  done  in  a 
program.  Therefore,  when  unexpected  outputs  are  obtained 
it  is  difficult  if  not  impossible  to  determine  their 
validity.  Equally  perplexing  is  inadequate  documentation  of 
subsequent  modifications  incorporated  into  the  model.  In 
many  cases  without  adequate  documentation  it  was   impossible 
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for  a  new  analyst  an!  or  programmer  to  use  or  modify  an 
existing  program.  Ultimately  such  models  have  to  be 
completely  rewritten  or  the  time  required  to  make  them 
usable  was  greatly  in  excess  of  the  time  required  when 
adequate  documentation  was  available.  In  a  cited  example 
inadequate  documentation  caused  an  agency  to  spend  over  a 
year  to  determine  how  a  particular  complex  model 
operated.  Another  example  indicated  the  difference  of  six 
man-months  of  work  incurred  because  of  the  lack  of 
sufficient  documentation.  Although  the  Comptroller  General 
found  it  difficult  to  determine  the  aggregate  cost  of 
increased  operating  expenses  due  to  inadequate 
documentation,  he  did  indicate  it  was  high  because  of  the 
number  of  cases  uncovered. 

Lack  of  sufficient  documentation  was  the  major  cause  of 
the  problems  encountered  by  the  author  during  the  conversion 
of  the  ATLAS  model.  Shen  machine  differences  required 
changing  the  program  code  there  was  minimal  guidance  as  to 
which  sections  of  the  code  corresponded  to  particular 
functions  described  in  the  user's  manual.  "  22  ]  Also, 
details  of  the  mathematical  structure  of  ATLAS  are  not 
contained  in  the  manual.  For  details  of  the  model 
structure  references  are  made  to  a  models  manual  [23]  for 
the  predecessor  of  ATLAS.  The  extent  to  which  the  structure 
of  this  model  has  been  incorporated  into  ATLAS  is  not 
explicitly  stated.  The  situation  is  further  complicated  by 
the  fact  that  pertinent  assumptions  basic  to  the  model 
formulae  are  not  in  the  models  manual;  they  are  in  the 
user's  manual  listed  in  a  haphazard  manner.  Since  the 
models  discussed  were  originally  designed  for  the 
predecessor  of  ATLAS  there  is  no  assurance  that  changes  to 
the  model  formulae  were  Dot  made  during  the  evolution  of 
ATLAS.  This  suspicion  is  enhanced  by  the  fact  that  in  1973, 
five  years  after  the  design  of  ATLAS,  a  significant 
programming  error  was  found.  [24] 
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There  are  a  myriad  of  reasons  why  model  documentation  is 
inadequate.  One  of  the  primary  causes  is  that  documentation 
guidelines  and  policies  are  developed  by  individual  Federal 
departments  and  agencies.  At  the  highest  levels  the 
guidance  is  necessarily  general,  as  it  moves  down  the 
organization  further  more  explicit  implementing  directives 
are  provided  culminating  in  directives  issued  by  the 
developing  agency.  Hence,  some  documentation  is  brief  and 
simplistic;  other  documentation  is  detailed,  voluminous  and 
complex;  neither  may  prove  to  be  adequate.  Adequacy  is 
determined  by  the  ability  of  other  than  the  originators  to 
use  and  understand  the  model.  The  Comptroller  General  found 
that  even  when  guidelines  and  standards  were  prescribed 
managers  of  modeling  projects  failed  to  insure 
compliance.  The  type  and  content  of  documentation  is  often 
decided  by  computer  technicians  or  ADP  operators.  [15] 
Shubik  [17]  has  cited  this  practice  as  unprofessional  since 
ambitious  programmers  have  been  known  to  change  coding  in 
the  pursuit  of  computing  efficiency  without  making  note  of 
the  fact. 

An  examination  of  264  model  documentation  packages  at  10 
California  installations  revealed  none  fully  complied  with 
the  agency  standards  and  most  were  incomplete,  inconsistant, 
and  inadequate.  [15]  In  most  cases  programmers  determined 
what  documentation  to  prepare  based  on  their  own 
judgement.  Managers  responsive  for  developing  models 
indicated  that  the  reason  standards  were  not  adhered  to  was 
because  of  time  contraints.  Completion  dates  frequently 
were  given  precedence  over  preparation  of  adequate 
documentation.  The  desire  to  complete  a  model  and  get  it 
operational  by  a  given  date  overrode  the  need  for 
documentation. 
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An  aspect  of  managerial  responsibilities  concerning 
documentation  is  that  it  be  kept  current.  Even  if  a  model 
is  initially  adequately  documented  it  will  eventually  be 
modified  and  its  documentation  must  be  likewise 
updated.  This  is  a  major  problem  because  it  requires 
diligence  to  update  documentation.  Siven  time  and  resource 
constraints  and  the  exigencies  of  the  decision  making 
process  it  is  a  task  that  can  easily  be  put  off  since  the 
model  will  work  without  such  documentation.  The  problem 
comes  later  when  the  personnel  that  modified  the  program 
become  involved  with  other  demanding  problems,  leave  or  are 
transferred.  Later  the  documentation  is  difficult  to 
prepare  because  the  reasons  why  or  how  the  modification  was 
made  become  unclear  or  those  that  knew  what  was  done  are  no 
longer  available. 

There  are  numerous  comment  cards  in  the  ATLAS  program 
that  indicate  changes  were  made.  However,  there  is  no 
formal  documentation,  with  one  exception,  to  indicate  what 
or  how  these  changes  were  made.  Informal  documentation 
provided  was  minimal  and  superficial  and  did  not  address  all 
the  indicated  changes.  The  one  exception  was  a  formal 
document  [25]  prepared  in  1974,  which  discussed  improvements 
in  the  treatment  of  barriers  and  personnel  replacements.  A 
global  variable  and  subroutine  listing  were  provided  but 
these  were  not  up  to  date;  had  they  been  current  the 
conversion  would  have  been  facilitated. 

Poor  or  nonexisting  documentation  persists  irrespective 
of  the  efforts  of  the  Comptroller  General  and  the  Department 
of  Defense.  Inspite  of  the  identification  of  this  problem 
early  in  this  decade  [1,10,15]  its  presence  continues  to  be 
a  problem  at  the  close  of  the  decade.  [3]  The  continuing 
lack  of  detailed  documentation  that  enables  an  analyst  to 
understand  what  goes  on  inside  a  model  is  cited   by   Shupack 
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[26]  in  1979,  as  a  limiting  factor  in  the  use  of 
theatre- level  combat  models.  Although  there  was  a 
noticeable  improvement,  he  found  the  level  of  documentation 
of  IDAGAM  not  sufficient  to  insure  the  easy  and  proper  use 
of  the  model  without  supplementing  the  available 
documentation.  Without  adequate  documentation  the  analyst 
is  apt  to  make  erroneous  conclusions  regarding  the  processes 
occurring  within  a  model. 

Although  documentation  problems  persist  genuiQe  attempts 
to  resolve  the  problems  have  been  made.  Current  combat 
modeling  efforts  at  the  Naval  Postgraduate  School  are  not 
only  using  languages  especially  designed  for  simulation  but 
the  documentation  methods  employed  enhance  the  transparency 
of  the  model.  For  examples  see  Ref.  27,  28,  29.  Other 
agencies  have  also  made  inroads  toward  improving  the 
adequacy  of  documentation.  See  Ref.  30,  31, 
32.  Irrespective  of  these  improvements  the  author  believes 
a  vital  aspect  of  documentation  is  being  overlooked.  This 
aspect  and  appropriate  recommendations  are  set  forth  in  the 
next  chapter. 


SUMMARY 


Studies  as  well  as  intuition  reveal  that  to  understand 
the  workings  of  a  complex  model  an  analysts  requires  a 
detailed  explanation  of  the  calculations  and  data 
manipulation  performed  by  a  simulation.  Implicit  and 
explicit  assumptions  and  inputs  must  be  known  if  an  analyst 
is  not  going  to  use  a  model  as  a  black  box.  A  detailed 
knowledge  of  the  variables,  subprograms  and  their 
relationships  must  be  acquired  if  a  model  is  to  be 
transferred  from  one  using  agency  to  another.  Rarely  will 
both  organizations  posses  the  same  brand  of  data   processing 
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equipment,  more  likely  than  not  there  will  be  great 
dissimilarities.  Conversions  from  one  machine  to  another 
requires  that  changes  be  made  to  the  model.  To  change  the 
model  the  programmer  and  or  analyst  must  know  the  effect  his 
change  will  have  not  on  only  the  particular  line  of  code  he 
is  changing  but  throughout  the  program.  Analysts  must  not 
only  know  how  a  change  will  affect  ths  physical 
minipulations  and  computations  of  the  various  parts  of  the 
model  but  how  it  will  interact  with  the  implict  and  explicit 
assumptions  of  the  model.  To  gain  this  level  of 
comprehension  of  a  model  designed  by  someone  else  the 
analyst  and  programmer  of  the  gaining  organization  must 
consult  the  documentation  provided  with  the  model. 
Without  such  documentation  analysts  have  chosen  to  construct 
a  new  model.  Unfortunately,  the  general  concensus  of  the 
investigations  was  that  documentation  was  of  dubious  quality 
and  generally  inadequate.  [1,10,15] 
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71.   PROPOSED  HIERARCHY  OP  DOCUMENTATION 


The  discussion  of  who  the  user  is,  the  role  of  the 
officer  analyst,  and  the  documentation  concept  that  follows 
is  based  on  seven  years  experience  in  operations  research 
related  assignments,  discussions  with  analysts,  programmers 
and  students,  and  problems  encountered  during  conversion  of 
the  ATLAS  model.  The  author  has  drawn  upon  the  above 
sources  and  selected  litsrature  [4,  15,  17,  33,  34,  35]  to 
propose  a  new  concept  in  model  documentation.  This  new 
concept  is  intended  to  refine  and  supplement  current 
documentation  methods.  Implementation  of  the  proposed 
concept  will  hopefully  fill  a  void  that  currently  exists  and 
will  greatly  improve  the  transparency  and  transferability  of 
combat  model. 


A.   THE  USEE  AND  THE  ANALYST 


Before  proceeding  further  it  is  necessary  to  define  the 
often  referred  "user".  It  is  one  of  the  more  vague  terms 
associated  with  combat  modeling. 

Now,  who  is  the  user?  The  user  is  the  study  director 
and/or  decision  maker.  These  individuals  need  documentation 
that  provides  an  overview  of  the  model,  with  indications  of 
its  capabilities  and  limitations  and  general  applicability 
to  the  problem  under  study.  The  details  of  the  conceptual 
basis  of  the  model  and  how  and  what  information  can  be 
provided  along  with  an  evaluation  of  its  suitability  should 
be  provided  by  the  analyst. 
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The  analyst  is  the  manipulator  of  the  model.  He  is 
responsible  for  using  the  model  to  support  the  study 
objectives.  This  means  he  is  responsible  for  the  collection 
and  insertion  of  input  data  and  interpretation  of  the  output 
data.  If  the  model  is  small,  he  may  do  the  coding;  or  if  it 
is  complex,  a  programmer  may  assist  him  in  the  coding 
process.  Any  interpretation  of  how  the  model  represents 
reality  is  the  analyst's  responsibility.  To  fulfill  this 
responsibility  he  must  be  familiar  with  the  conceptual  basis 
of  the  model,  its  underlying  assumptions,  how  the  model  was 
transformed  into  the  program  code,  and  how  the  cole  operates 
to  present  the  process  modeled.  The  analyst  must  know  if 
any  changes  to  the  conceptual  foundation  of  the  model 
occurred  in  the  process  of  programming  (coding) . 

There  is  a  reluctance  on  the  part  of  many  military 
analysts  to  gain  an  intimate  understanding  of  a  complex 
combat  modal.  Exingencies  of  the  organization  are  some  of 
the  prime  reasons.  In  the  daily  demands  to  manipulate  a 
model  and  to  produce  results  there  is  insufficient 
allocation  of  time  to  study  the  inner  workings  of  the 
model.  Reliance  is  placed  almost  exclusively  on  the  user's 
manual  to  explain  the  results  produced. 

Some  analysts  (this  is  particularly  true  of  the  officer 
analyst)  refuse  to  gain  detailed  understanding  of  the  inner 
workings  of  a  model  because  they  do  not  perceive  that  to  be 
their  function.  Their  perception  is  that  they  are  the  user 
and  do  not  require  that  level  of  detailed  knowledge.  Others 
hesitate  to  learn  or  become  well  versed  in  the  functions  of 
a  particular  model  because  they  fear  such  expertise  will 
label  them  and  hence  limit  the  scope  of  their  future 
assignments.  In  discussions  with  officer  analyst  students 
and  practicing  officer  analysts,  the  author  found  these 
attitudes  to  be  quite  pervasive.  Many  considered  detailed 
knowledge   of  how  a  model  operated  to  be  in  the  realm  of  the 
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programmer's  responsibility.  Little  or  no  consideration  was 
given  to  the  fact  that  most  programmers  have  little  or  no 
military  background  and  cannot  possibly  relate  the 
machinations  of  the  program  code  to  the  combat  process.  The 
programmer  views  proper  operation  in  terms  of  proper 
execution  of  programmed  instructions.  Whether  or  not 
instructions  properly  depict  the  realities  of  the  combat 
process  are  beyond  their  scope  of  interest.  They  have 
neither  knowledge,  inclination  or  experience  to  evaluate  the 
fidelity  of  the  code. 

Attitudinal  attributes  of  military  analysts  can  be 
understood  in  light  of  their  rapid  turnover  in 
assignments.  Rapidity  of  reassignment  discourages  the 
desire  •  to  gain  detailed  knowledge  of  any  particular 
model.  Most  likely  they  will  be  reassigned  shortly,  to 
other  type  duties.  If  subsequent  assignments  deal  with 
combat  models,  most  likely,  it  will  be  a  different 
model.  Any  time  expended  in  the  study  of  details  of  the 
model  associated  with  the  current  assignment  is  considered 
of  minimal  value.  When  an  effort  is  made  to  understand  a 
model  it  is  often  frustrated  by  inadequate  documentation. 

Probably  the  most  frequent  attitude  seen  in  officer 
analysts  is  the  one  associated  with  the  military  psyche, 
that  of  being  a  generalist.  This  attitude  is  the  result 
of  the  total  experience  of  the  profession;  it  has  been  the 
way  to  reach  success  in  the  past,  and  many  believe  that  it 
still  is  the  way  to  success.  Although  there  has  been 
considerable  effort  to  change  this  mind  set,  the  example  of 
the  last  few  centuries  is  difficult  to  overcome.  Compared 
to  the  existence  of  the  military  profession,  the  experience 
of  the  military  analyst  is  of  recent  vintage.  Only 
favorable  experience  will  provide  the  impetus  for  change. 
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The  transfer  of  models  between  agencies  will  contribute 
favorably  toward  convincing  analysts  to  gain  detailed 
knowledge  of  combat  models.  If  they  know  that  the  effort 
expended  now  can  be  used  at  a  later  time  in  some  future 
assignment,  the  effort  will  be  considered  worthwhile.  But 
adequate  documentation  is  a  prerequisite  of  such 
transferability.  Hence,  there  is  an  intecdependency 
betweeen  documentation  and  convincing  military  analysts  to 
gain  a  detailed  understanding  of  combat  models.  If  they  can 
see  value  in  the  expenditure  of  tie  time  and  effort,  they 
will  be  willing  to  make  the  commitment. 

Shubik  [16]  has  found  that  the  mathematical  modeler  and 
the  person  who  understands  the  reality  the  model  attempts  to 
abstract  are  not  necessarily  the  same  person.  The  combat 
process  is  best  understood  by  senior  military  decision 
makers  who  are  generally  unable  to  translate  it  into  the 
appropriate  abstraction  and  who  generally  desire  greater 
detail  than  is  necessary.  A  good  model  is  one  that  is  able 
to  abstract  and  describe  Dnly  that  which  is  relevant  to  the 
problem  under  investigation.  On  the  other  hand  the 
mathematical  modeler  is  frequently  an  individual  who 
generally  lacks  the  experience  and  an  appreciation  of  the 
nuances  of  combat  which  can  lead  to  the  development  of  an 
ill-structured  model.  The  military  officer  analyst 
represents  a  step  toward  providing  a  modeler  or  model 
operator  who  not  only  understands  the  mathematical  aspects 
but  the  military  factors  as  well.  If  this  capability  is  to 
be  maximized  when  dealing  with  existing  models,  there  must 
be  documentation  which  allows  the  analyst  to  link  the 
conceptual  model  formulation  to  the  executable 
program.  Then  he  can  understand  the  conceptual  basis  of  the 
model,  insure  the  program  fulfills  the  concept,  and  act  as  a 
source  of  information  for  the  decision  maker. 
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B.   ORIGIN  OF  THE  DOCU HEN! ATION  SHORTFALL 


The  initial  theatre- level  combat  models  used  in  the 
United  States  were  designed  and  operated  by  contract 
organizations  (e.g.  Research  Analysis  Corp.).  These  models 
had  a  manual  referred  to  as  a  user's  manual  which  provided 
an  overview  of  the  model,  input  requirements,  and  a 
description  of  the  output.  For  an  example  see  Ref.  22. 
Documentation  within  the  manual  was  superficial,  at  most  it 
provided  the  proverbial  big  picture.  Sufficient 
information  was  provided  to  determine  the  general 
capabilities  of  the  model,  possible  suitability  for  a 
particular  study  and  a  general  indication  of  expected 
outputs.  It  provided  little  if  any  information  on  the 
insights  that  could  be  attained  or  subtleties  of  the 
model.  The  user  for  whom  this  "user's  manual"  was  designed 
was  a  project  manager  or  some  level  of  decision  maker.  The 
details  could  be  filled  in  by  the  persons  who  would  operate 
the  model,  this  most  likely  was  the  designer  of  the 
model.  Because  of  budgetary  reasons  and  doubts  that  these 
model  operaters  understood  the  nuances  of  military 
operations,  the  decision  was  made  to  bring  these  models 
directly  under  Dept.  of  the  Army  control. 

When  these  models  passed  "to  Army  control  all 
documentation  passed  with  them.  Unfortunately,  in  most 
cases  the  documentation  was  minimal  consisting  primarily  of 
program  listings,  operator  instructions,  global  variable  and 
subroutine  listings,  flowcharts,  user's  manual  and  possibly 
some  limited  discussion  of  the  formulation  of  the  model. 
These  models  were  placed  under  control  of  the  agency  that 
would  use  them  in  support  of  force  planning  and  strategy 
studies.    The  designated  users  that  inherited  these   models 
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were  the  military  analysts  (officers  and  federal  employees) 
assigned  to  the  organization  that  rscaived  the  models.    The 
analysts   then   took   the   model   and   the  user's  manual  and 
proceeded  to  exercise  the   model  in   support  of   on-going 
projects. 

Use  of  the  user's  maaual  by  the  analyst  was  mandated  by 
the  fact  that  input  requirements  and  preparation  were 
contained  in  it.  However,  when  the  models  were  initially 
transferred  from  the  designing  corporation  to  the  Army  no 
supplemental  documentation  was  prepared  or 
provided.  Analysts  were  using  a  manual  that  provided  only  a 
superficial  examination  of  the  model.  They  performed 
analyses  using  a  manual  originally  designed  for  a  study 
director  or  decision  makar.  These  "user's  manuals"  did  not 
contain  the  detailed  information  of  the  model  that  is 
necessary  if  the  analyst  is  to  know  and  understand  the 
machinations  through  which  the  input  data  is  exposed  to 
produce  a  given  set  of  outputs. 

Evenually  conflicting  results  were  obtained  from  models 
supposedly  examining  the  same  situation.  When  called  upon 
to  resolve  or  explain  thase  discrepancies  the  analysts  were 
unable  to  do  so.  To  explain  the  conflicts  a  detailed 
understanding  of  the  coaceptual  basis  of  the  model  as  well 
as  detailed  information  of  the  translation  of  tha  conceptual 
model  into  program  code  was  needed.  Only  with  this 
information  could  the  analyst  explain  why  two  models 
examining  the  same  combat  process  under  the  same  conditions 
produced  different  results. 

Analysts  then  discovered  that  the  documentation  provided 
with  the  model  did  not  provide  this  insightful 
information.  To  answer  the  question  the  model  dasigner  had 
to  be  contacted.  At  times  this  was  impossible  because  the 
actual   model   the  designer  no  longer  was  with  the  firm  that 
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developed  the  model,  nor  was  there  available  any 
supplemental  information  on  the  inner  workings  of  the  model. 
Instances  in  which  the  designer  could  be  located  usually 
proved  just  as  unrewarding.  In  most  cases  the  designer  had 
no  supplemental  formal  documentation.  When  he  acted  as  the 
operator  of  the  model  on  behalf  of  the  Army  he  could  answer: 
such  questions  based  on  his  design  knowledge  and  daily 
contract  with  the  modal.  Since  there  had  been  no  formal 
requirement  for  such  supplemental  information,  it  never  was 
prepared.  Once  the  model  passed  under  operational  control 
of  the  Army,  the  designer  was  precluded  from  daily 
contact.  Over  time  the  designer's  intimate  familiarity  with 
the  model  waned,  especially  the  intracacies  of  the 
translation  from  basic  concept  to  operating  program 
code.  If  the  designer  had  not  done  the  actually  coding, 
similar  results  were  usually  experienced  when  trying  to 
locate  the  original  programmer  or  obtain  supplemental 
information.  What  the  program  code  was  actually  doing  was 
not  readily  apparent  and  documentation  or  personal  knowledge 
of  its  operation  were  unavailable.  The  situation  is  best 
expressed  by  a  quote  from  Robert  Frost.  When  once  asked  by 
a  critic  what  he  meant  by  a  particular  phrase  in  one  of  his 
works,  he  replied,  "When  I  wrote  it  Sod  and  I  knew  what  it 
meant,  now  only  God  knows".  This  same  situation  will  also 
prevail  with  combat  models  unless  adequate  documentation  is 
created  concurrent  with  the  design  and  development  of  the 
model. 


COMMUNICATION  AND  LEVELS  OF  PARTICIPATION 


The  prime  purpose  of  documentation  is  communication: 
communication  of  why  and  how  realities  are  abstracted  and 
condensed  into  a  form  suitable  for  exercise  on  a  computer  to 
predict  a  future  state  of  some   combat   process.    Generally 
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the  level  of  professional  communication  between  decision 
makers,  analysts,  and  modelers  has  been  low  as  attested  by 
the  credibility  problem  previously  discussed.  This  coupled 
with  the  turnover  of  military  decision  makers  and  analysts 
has  not  enhanced  conpr ehension  of  and  control  over 
large-scale  combat  models.  Very  often  the  information  about 
context  and  limitations  of  these  models  has  not  been  fully 
understood  by  the  military  analyst  and  therefore  not  clearly 
presented  to  tha  decision  maker.  Documentation  that  can 
provide  such  enlightenment  to  the  analyst  will  be  a 
significant,  step  toward  providing  the  required  insights  into 
combat  models  needed  by  decision  makers.  Through  adequate 
documentation  ths  user  and  analyst  gain  understanding,  and 
with  understanding  can  come  acceptance  and  the  decision  to 
use  the  particular  model. 

In  combat  modeling  there  ara  three  levels  of 
participation  with  necessary  intercommunication.  At  the 
highest  level  is  the  decision  maker  who  uses  the  model  as  an 
aid  to  gain  insights  and  in  conjunction  with  judgement 
arrive  at  a  decision  concerning  force  structure  or 
strategy.  The  intermediate  level  is  occupiad  by  the 
analyst,  who  either  designs  a  model  or  uses  an  existing 
model;  prepares  inputs;  exercises  the  model  and  interprets 
the  outputs.  If  the  model  is  new  to  the  decision  maker  he 
also  aids  in  the  model  selection  process  by  recommending  and 
explaining  the  inherent  attributes  of  particular  models  in 
pursuit  of  a  projected  study.  At  the  lowest  level  is  the 
programmer,  who  writes  the  necessary  code  during  model 
construction,  explains  the  limitations  of  execution  of  a 
model  on  a  particular  computer,  insures  the  model  is 
processed  as  the  program  intended  it  and  to  code  necessary 
modifications  to  a  nodel.  Each  level  has  unique 
requirements  and  responsibilities  and  therefore  the 
documentation  required  by  each  is  unique. 
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The  decision  maker  uses  his  documentation  to  gain  an 
overall  appreciation  of  a  nodel  and  sufficient  understanding 
to  question  the  analyst  on  model  details  necessary  to  make  a 
decision  to  employ  the  model  and  use  its  outputs.  It 
provides  a  means  of  conducting  discussions  that  can  reveal 
what  insights  can  be  gained  from  a  particular  model.  The 
decision  maker  uses  his  documentation  as  a  link  between  the 
analyst  and  himself. 

The  analyst  occupies  the  pivotal  position  between  the 
decision  maker  and  the  programmer.  His  documentation  is 
necessarily  the  most  broad  in  scope  as  well  as  variation  of 
detail.  It  must  provide  intimate  knowledge  of  the  model  to 
allow  selection,  sufficient  detail  to  answer  questions  from 
the  decision  maker  and  ask  questions  of  the  programmer.  It 
must  provide  the  key  to  the  inner  workings  of  the  model. 

Programmer  documentation  allows  the  programmer  to 
maintain  and  modify  the  model  and  answer  the  questions  of 
the  analyst.  It  allows  the  programmer  to  maintain 
efficiency  of  operation  and  insure  execution  in  accordance 
with  the  dictates  of  the  design.  It  must  clearly  indicate 
how  the  computer  interacts  the  inputs  and  programming  code 
to  produce  the  outputs. 


D.   A  CONCEPTUAL  THEORY  OF  MODEL  DOCUMENTATION 


Current  texts  that  describe  the  documentation  process 
usually  are  texts  on  the  subject  of  computer  systems.  The 
guidance  provided,  though  attempting  to  be  general,  is 
oriented  toward  documentation  of  programs  that  are  bulk 
processors  of  systematic  information,  i.e.  financial 
records,  administrative  records  etc.  Most  emphasis  is 
placed   on   how   to  use  the  available  program  to  process  the 
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data  available,  minimal  emphasis  is  directed  toward  how  the 
processing  is  accomplished,  because  the  processing  is  not 
complex  but  routine  and  easily  understood.  For  an  example 
see  Ref .  33. 

In  the  field  of  combat  modeling  little  guidance  is 
available  in  the  form  of  textbooks  describing  documentation 
methodologies.  In  order  to  maximize  detailed  unlerstanding 
as  well  as  transferability  of  combat  models  a  new  concept  of 
documentation  is  proposed  and  set  forth  in  this  section. 

To  fulfill  the  requirements  discussed  above  the 
documentation  must  provide  the  complete  understanding  of  how 
the  model  was  created,  underlying  assumptions,  explicit 
description  of  the  formulations  and  numerical  methods 
employed  and  detailed  description  of  the  mechanics  of  the 
program  code  and  its  execution.  Most  current  documentation 
described  as  a  user's  manual  (e.g.  see  Ref.  22),  executive 
summary  or  by  some  similar  title  is  sufficient  to  fulfill 
the  proposed  non-technical  documentation  that  follows. 
Programmer's  documentation  is  widely  described  in  the 
literature  of  computer  systems  and  needs  just  a  few 
additions  in  respect  to  combat  models.  Assuming  that  the 
documentation  is  provided  as  described  in  the  Literature, 
minimal  modification  is  required.  Primarily  this  is  a 
charting  and  cataloging  procedure  to  allow  ease  of  tracing 
variables  through  the  program  and  understanding  the  linking 
of  subprograms  to  the  main  program.  Current  documentation 
requirements  stress  flowcharting.  Flowcharting  is 
comparable  to  electronic  schematics  and  allows  one  to  see 
the  logical  flow  of  the  process  being  programmed.  In  the 
coding  step  this  logical  flow  is  translated  into  a 
mechanical  flow  of  variables  through  mathematical  and 
logical  formulations  and  subprograms.  A  charting  procedure 
will  be  like  a  blueprint  that  shows  the  physical  connection 
of   the   main   and  subprograms.   The  catalog  will  explicitly 
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state   how   variables   are   interacted   and   passad   between 
subprograms. 

For  the  analyst's  documentation  a  new  concept  is 
proposed.  Rather  than  describing  the  modal  in  the 
traditional  manner,  i.e.  from  the  context  of  justification, 
the  analyst's  documentation  should  be  presented  from  the 
context  of  discovery.  [2]  Using  the  traditional  concept  the 
model  is  documented  by  stating  the  assumptions  of  premises 
which  determine  the  outputs  of  the  model,  showing  the  final 
mathematical  formulae  that  represent  the  abstractions  of  the 
relavent  characteristics  of  the  modeled  process,  discussing 
the  inputs  required  to  support  the  given  formulae,  and 
listing  the  outputs  that  :an  be  obtained  from  the  model  and 
inputs. 

This  is  not  the  manner  in  which  the  model  was  formulated 
and  such  documentation  in  the  context  of  justification  does 
not  enlighten  the  analyst  in  terms  of  how  the  designer 
arrived  at  this  particular  abstraction  of  the  combat 
process.  One  must  conclude  that  this  type  of  documentation 
is  not  of  great  help  to  the  inexperienced  analyst  if  he  is 
attempting  to  understand  the  essence  of  a  combat  model.  If 
the  analyst  is  to  gain  a  intimate  knowledge  of  the 
functioning  of  a  model,  ha  must  be  able  to  ascertain  how  the 
model  came  to  be.  This  provides  him  two  invaluable 
insights.  First,  he  gains  an  insight  into  the  abstraction 
process  of  the  designer.  It  is  a  glimpse  of  tha  reasoning 
procsess  by  which  the  designer  cut  through  the  myriad  of 
details  to  deduce  the  fundamental  variables  that  allow  a 
model  to  approximate  the  complexity  of  a  given  combat 
process  without  having  to  model  each  of  the  multitudinous 
factors  that  compose  the  actual  process.  Second,  he  gains 
knowledge  of  the  factors  that  were  considered  as 
representative  of  the  coabat  process  but  were  discarded 
because   they   seemed  not  to  be  necessary  predictors.   This 
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step  in  itself  is  worth  the  cost  of  providing  this  type  of 
documentation.  What  it  provides  is  a  historic  record  of 
factors  and  related  assumptions  considered  and  reasons  why 
they  were  ultimately  rejected  or  accepted.  The  information 
provided  must  be  succinct  and  allow  understanding  of  the 
abstraction  process  without  introducing  voluminous  detail 
and  unnecessary  costs.  This  will  be  an  aid  not  only  for 
the  follow-on  analyst  bat  all  those  that  come  later.  It 
will  preclude  subsequent  analysts  from  expending  time  and 
money  to  determine  why  particular  factors  are  or  are  not 
modeled.  Hence,  if  they  wish  to  modify  or  elaborate  upon  a 
combat  model  and  information  about  previously  considered  and 
rejected  alternatives  is  available,  the  expenditure  of 
resources  to  reinvestigate  a  particular  alternative  and  gain 
only  duplicate  information  will  be  precluded. 


E.   REQUIRED  LEVELS  OF  DOCUMENTATION 


All  the  evidence  gathered  indicates  three  levels  of 
documentation  for  models  are  required.  These  levels  of 
documentation  are: 

(1)  decision- maker  level,  (facilitates  communication 
between  the  decision  maker  and  analyst) ; 

(2)  analyst  level  (fosters  communication  between  analyst 
and  decision  maker  and  enhances  working  relationship  with 
the  programmer) ; 

(3)  programmer  level  (provides  detailed  understanding  of 
program  code  and  facilitates  communication  with  the 
analyst) . 
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First,  the  decision  maker  who  uses  the  model  as  an  aid 
to  his  judgement  requires  a  non- technical  description  of  the 
model.  Non-technical  not  because  the  decision  maker  cannot 
comprehend  the  technical  details  but  because  the  exigencies 
of  the  organization  preclude  him  from  devoting  the  necessary 
time  and  effort.  Second,  the  analyst  that  exercises  the 
model,  prepares  the  analysis,  and  assists  the  decision  maker 
requires  a  technical  description  of  the  model.  This  enables 
the  analyst  to  know  how  the  model  functions  and  explain  and 
interpret  the  outputs  of  the  model.  Third,  the  programmer 
needs  detailed  documentation  that  explains  the  mechanics  of 
the  program.  This  documentation  provides  the  means  to 
troubleshoot  problems  encountered  during  routine  running  of 
the  model  and  implement  future  modifications.  The  levels 
of  activity  and  their  required  documentation  are  shown  at 
Fig  2. 


F.   DECISION  MAKER'S  NON-TECHNICAL  DOCUMENTATION 


A  non-technical  reference  manual  for  use  by  decision 
makers  should  provide  sufficient  information  to  determine 
general  applicability  of  the  model.  It  should  include  a 
general  description  of  how  the  model  operates  and  the  major 
components,  the  specific  purpose  for  which  the  model  was 
originally  designed  and  the  known  limitations  should  be 
stated.  This  will  allow  the  decision  maker  to  assertain  the 
overall  ability  of  this  particular  model  to  assist  him  in 
the  problem  at  hand.  The  manual  should  describe  the  data 
requirements,  available  outputs,  and  any  options  provided  by 
the  model.  This  facilitates  the  initial  planning  process 
because  it  provides  a  basis  for  estimating  the  expected  time 
to  gather  the  input  data  and  what  to  expect  in  the  form  of 
outputs.   Physical  limitations,  such  as  computers  for   which 
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usable  versions  of  the  model  are  available,  the  programming 
language  used,  and  typical  running  times  for  the  model,  are 
necessary  contents  of  the  non-technical  manual.  The  time  of 
such  senior  managers  is  limited  and  valuable.  Only  that 
information  necessary  to  provide  a  general  description  and 
basis  for  meaningful  discussions  between  the  decision  maker 
and  the  analyst  should  be  included.  Any  details  required 
will  be  provided  by  the  analyst,  who  will  conduct  the  study, 
during  planning,  progress,  and  review  sessions.  Yet  the 
documentation  must  make  the  model  sufficiently  transparent 
so  that  the  decision  maker:  understands  what  is  happening  and 
finds   the   model   credible  for    the    problem  at   hand. 


G.       ANALYST'S    CONCEPTUAL-TECHNICAL    DOCUMENTATION 


The  analyst's  conceptual-technical  reference  manual  must 
necessarily  contain  detailed  information  on  all  aspects  of 
the      model.  This      is      the      document      that   determines   the 

overall  worth  of  the  model  as  an  analytical  tool.  If 
sufficient  detail  is  contained  then  the  analyst  can  become 
intimately  familiar  with  the  model,  so  that  any  aberation 
encountered  during  operation  of  the  model  can  be  understood 
by  him  and  explained  to  the  decision  maker.  To  maximize  its 
value  to  an  analyst,  it  should  be  written  by  the  analyst  or 
analysts  that  design  the  model.  Their  professional 
experience  will  guide  them  in  providing  the  kind  of 
information  about  the  model  that  they  would  want  if  they 
were   using    a   model   designed    by   someone    else. 

The  size  of  the  analyst's  reference  manual  will  be  a 
function  of  the  complexity  of  the  model.  To  be  useful  and 
credible  a  model  must  be  transparent  and  transferable.  To 
be  transparent  and  transferable  all  necessary  details  of  the 
model   must   be   provided.      Information   on  the   data      base      used 
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in  the  model,  input  requirements  and  format  as  well  as 
output  format  and  options  must  be  detailed.  All  that  can  be 
assumed  is  that  the  analyst  is  knowledgeable  in  the  use  of 
the  tools  of  his  profession.  In  preparing  tha  analyst's 
manual  no  prior  knowlelga  of  the  model  can  be  assumed.  As 
Morris  [2]  recommended,  the  obvious  should  be  written  down. 
All  constraints  and  limitations  must  be  described  in  detail 
as  well  as  assumptions  used,  logic  flow  and 
interactions.  Sufficient  technical  detail  to  allow  the 
analyst  to  manually  trace  inputs  through  the  algorithms  is 
necessary.  Mathematical,  statistical,  an!  numerical 
methods  incorporated  in  the  model  should  be  described 
including  any  new  or  unique  applications.  Any  constraints 
which  will  affect  the  accuracy  of  the  modeL  must  be 
identified.  Obvious  pitfalls  must  be  stated;  thay  are  only 
obvious  to  the  developer  and  in  complex  models  without 
documentation  they  can  even  be  forgotten  by  the 
developer.  The  physical  processes  simulated  must  be 
described  including  an  explanation  and  rationalization  of 
the  techniques  used.  Eaoh  variable  and  the  entity  it 
represents  must  be  clearly  stated.  Lastly,  sufficient 
instructions  describing  how  to  set  up  and  use  the  model  and 
flow  charts  keyed  to  the  program  instructions  should  be 
provided.  Squally  important  is  a  system  that  keys  the 
description  of  each  mathematical  formulation  in  the  manual 
to  the  appropriate  section  and  lines  of  the  program 
code.  This  will  facilitate  the  location  of  tha  code,  when 
the  analyst  wants  to  modify  the  model. 

These  are  the  basic  requirements  for  the  contents  of 
documents  to  be  prepared  by  the  designer  of  a  model.  Each 
time  a  modification  to  the  model  is  made,  the  changed 
program  instruction,  reasons  why  the  change  was  necessary, 
how  and  what  is  affected  in  the  model,  and  the 
rationalization  for  the  particular  modification  should  be 
documented    and    incorporated    into     the     original 
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documentation.  This  aspsct  of  documentation  is  critical 
because  analyst  and  programmers  have  been  known  to  make 
small  insignificant  changes  to  programs  without  documenting 
them.  Two  results  can  come  about  because  of  this.  Over 
time  these  minor  undocumented  changes  accumulate,  then  one 
day  those  who  implemented  the  changes  transfer  or  depart. 
The  newly  arrived  analyst  now  has  a  model  which  does  not 
quite  fit  his  documentation.  If  enough  time  has  passed 
even  those  who  made  the  change  will  not  completely  remember 
it  when  they  are  questioned  or  they  are  no  longer 
available.  Even  more  disheartening  is  that  in  failing  to 
document,  a  critical  inspection  of  the  effect  of  the  change 
on  other  parts  of  the  model  is  not  made.  Unbeknownst  to  the 
individual  the  modification  causes  an  effect  elsewhere  in 
the  model  that  is  not  readily  .  discernable  at  the 
time.  Utimately,  another  modification  is  made  and  the  model 
malfunctions  or  an  anomalous  output  occurs.  If  undocumented 
modifications  have  been  made  and  forgotten  it  may  be 
impossible  to  trace  the  cause  of  the  anomaly.  If  it  can  be 
traced,  the  cost  of  trouble  shooting  and  correcting  the 
model  will  be  greater  than  the  cost  of  documenting  the 
modifications  at  the  time  of  their  addition.  Current 
studies  will  be  delayed  with  their  attendant  costs  and  the 
credibility  of  the  modeling  community  will  suffer.  The 
decision  maker  justifiably  becomes  skeptical  when  the 
analyst  cannot  explain  why  the  results  of  a  model  are  not 
compatible  with  the  decision  maker's  intuitive 
expectations.  If  a  model  is  to  aid  in  the  decision  making 
process  its  outputs  must  be  explainable  and  understandable 
to  the  decision  maker.  If  the  analyst  has  not  designed  the 
model  then  his  only  source  of  the  necessary  information  is 
the  documentation  supplied  with  the  model. 
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H.   PROGRAMMER'S  TECHNICAL  DOCUMENTATION 


The  final  level  of  required  documentation  is  the 
detailed  information  reguired  by  the  programmer  for  the 
operation  and  maintenance  of  the  model.  This  is  the 
documentation  that  is  discussed  in  great  detail  in  all 
standard  texts,  while  methods  of  documenting  the  conceptual 
basis  of  models  is  almost  entirely  foregone.  The  manual  is 
necessarily  prepared  by  the  model  programmer  in  conjunction 
with  the  analyst  who  has  designed  the  model.  It  must 
contain  descriptions  of  each  global  variable  and  in  which 
subprograms  the  variable  is  found.  Descriptions  of  the 
functions  performed  by  the  program,  data  flow  charts, 
function  flow  charts,  approximations  and  numerical 
procedures  used  are  also  contained  in  this 
documentation.  Likewise,  any  implicit  assumptions  made  by 
the  programmer  during  the  coding  process  to  facilitate 
computation  must  be  documented. 

Often,  though  the  analyst  designer  has  structured  the 
model  to  handle  the  general  case,  during  implementation  of 
the  model  on  tha  computer  some  loss  of  generalization 
occurs.  In  the  search  for  programming  efficiency  the 
computer  programmer  may  oode  a  combat  process  so  that  the 
model  works  for  only  those  circumstance  explicitly  stated  by 
the  designer.  Whereas  the  designer  believes  he  has  a 
general  model  the  programmer  has  reduced  the  generality  of 
the  model.  Unless  this  is  documented  by  the  programmer,  in 
subsequent  use  of  the  model  the  fact  that  the  model  was 
programmed  in  such  a  manner  may  be  forgotten.  In  this  case 
the  computer  routines  embody  more  assumptions  than  exist  in 
the  formal  model  designed  by  the  analyst.  It  is  incumbent 
upon  the   programmer  to  bring  this  fact  to  the  attention  of 
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the  analyst.  He  inturn  Bust  include  this  information  in  his 
documentation.  Dnly  he  can  conceptualize  in  terms  of  real 
world  factors  the  impact  of  the  programmer's  assumptions  on 
the  inputs  and  outputs. 

Further  contents  of  the  detailed  programming  na,nual  must 
indicate  primary  and  secondary  storage  requirements,  error 
detection  and  recovery  procedures,  and  instructions  for 
initiation  and  termination  of  program  operation.  These 
pieces  of  information  are  required  if  other  programmers  are 
to  be  able  to  understand  and  operate  the  model.  If  a  model 
is  transferred  this  information  is  necessary  to  implement 
the  model  on  a  computer  of  different  design.  It  provides 
the  receiving  programmer  the  necessary  information  to 
prepare  system  operating  programs  and  procedures  so  that  the 
model  can  be  exercised  by  the  receiving  agency  without 
inadvertent ly  making  changes  to  the  logic  of  the 
model.  This  is  especially  critical  for  complex 
models.  With  such  models  attempts  to  restructure  programs 
to  overcome  incompatibilities  in  storage  or  other  machine 
requirements  may  introduce  changes  to  the  fundamental  logic 
of  the  model  unbeknownst  to  either  the  programmer  or  the 
analyst.  Without  adequate  documentation  such  restructuring 
must  be  accomplished  with  only  luck  to  guide  the  way  and  as 
complexity  of  the  model  increases  luck  is  quickly 
depleted . 

When  the  original  programmer  completes  the  programming 
(coding)  of  the  model  the  model  must  be  verified  and 
validated  in  conjuntion  with  the  analyst  designer.  When 
both  are  satisfied  that  the  model  is  operating  as  designed  a 
listing  of  the  compiled  assembled  program  should  be 
made.  This  is  then  supplemented  by  a  set  of  working  notes 
on  program  operations,  set  up  of  the  deck  to  exercise  the 
model,  special  programming  features  and  identification  of 
potential  problem  areas  and  suggested  solutions. 


63 


The  compilation  of  the  above  documentation  into  a 
detailed  programmer's  manual  will  provide  a  sound  basis  for 
review  and  analysis  of  the  model's  capabilities  and  for 
maintenance  of  the  model's  logic;  it  will  facilitate 
transfer  and  insure  transparency.  Furthermore,  each  time 
the  programmer  makes  a  modification  to  the  program  it  must 
be  documented  and  incorporated  into  the  programmer's 
manual.  Concurrently,  it  must  be  brought  to  the  attention 
of  the  analyst  to  relate  the  programming  change  to  the 
combat  process  modeled  and  update  the  analyst's  and  decision 
maker's  manual.  This  coordination  between  the  programmer 
and  the  analyst  must  always  occur  if  both  are  to  remain 
fully  cognizant  of  what  as  well  as  how  the  model  simulates 
the  real  world.  Only  through  such  documentation  can  a  model 
be  transferred  to  a  new  programmer  and  analyst  team  and  be 
knowledgeably   used. 


I.       INLINE    DOCUMENTATION,    A     MAP    THRU    THE    MAZE 


A  necessary  adjunct  to  this  proposed  documentation 
package  is  documentation  within  the  program.  Such 
documentation  should  be  in  the  form  of  comment  oards  that 
briefly  describe  the  main  program,  the  major  sub-models, 
suoroutine  and  each  function  subprogram.  They  should  be 
inserted  at  appropriate  locations  so  that  logic  of  the 
program  is  readily  apparent.  Nothing  is  more  frustrating 
to  an  analyst  that  is  attempting  to  gain  a  quick  basic 
understanding  of  a  model  than  a  series  of  calls  to 
subroutines  which  inturn  call  other  subroutines  and  or 
function  subprograms.  In  a  complex  model  this  quickly  hides 
the  logic  of  the  model  from  the  analyst.  To  pierce  this 
shroud  the  analyst  must  devote  time  that  could  better  be 
used  elsewhere.  If  the  exigencies  of  the  situation  demand  a 
rapid   response   and    no    other    model    is   readily  available      this 
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encourages  the  analyst  to  use  the  model  as  the  proverbial 
"black  box".  This  neither  adds  to  transparency  nor  does  it 
enhance  the  reputation  of  the  analyst  or  operations 
research . 

Most  current  programming  texts  recommend  using  comment 
cards  to  enhance  understanding  of  programs.  In  complex 
models  this  is  not  a  nicety  but  a  necessity.  Time  devoted 
to  this  effort  initially  not  only  enhances  the  use  of  a 
complex  model  but  reduces  recurring  costs.  Professionalism 
demands  that  a  model  should  not  be  used  without 
understanding  how  it  functions  or  at  least  believing  one 
understands  how  it  functions.  The  realities  of  military 
personnel  policies  dictate  that  military  analysts  will 
rotate  rapidly  through  assignments.  If  a  program  is  not 
documented  internally  this  self-educating  step  occurs  each 
time  a  new  analyst  uses  a  particular  model.  Given  a  complex 
model  and  normal  personnel  turnover  a  sizeable  cost  is 
incurred.  With  adequate  internal  documentation  this  cost 
will  occur  only  once.  Because  of  personnel  rotation  models 
not  currently  having  such  internal  documentation  should  have 
it  added.  Though  it  will  detract  an  analyst  for  an 
additional  amount  of  time  initially,  over  the  longer  term  a 
savings  will  be  realized.  Each  subsequent  analyst  will  not 
have  to  start  from  scratch  when  trying  to  gain  an 
understanding  of  how  the  model  functions;  understanding  will 
be  facilitated  by  internal  documentation  via  comment   cards. 


J.   UPDATING  A  MODEL'S  DDCO MENTATION 


Changes   to  models  occur  over  extended  periods  of  time. 
Obviously,  at  the  time  changes  are   instituted   the   process 
through   which   they   were   developed,  why  and  how  they  were 
entered  into  the  model  ani   all   assumptions   used   must   be 
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annotated  in  all  appropriate  documentation  at  the  agency 
instituting  the  change.  Some  of  these  changes  will  be 
extremely  large  and  will  justify  formal  changes  being 
printed  and  distributed  for  all  affected 
documentation.  Some  changes  will  be  small;  formal  printing 
of  individual  changes  would  not  be  justified.  These  would 
have  to  be  accumulated  and  changes  printed,  when 
economically  practical.  The  management  of  the  process  and 
the  determination  of  whan  printing  and  distribution  of 
changes  is  economically  practical  are  beyond  the  scope  of 
the  thesis.  rhey  are  important  subjects  and  should  be 
examined  in  some  future  thesis. 


K.   SUMMARY 


Since  the  analysts  and  programmers  that  develop  a 
particular  model  are  not  always  present  or  available  when 
the  model  is  exercised,  especially  if  it  is  transferred  to 
another  agency,  it  is  their  responsibility  to  detail  their 
assumptions,  simplifications  and  methodologies  and  provide 
evidence  that  the  rationale  behind  their  approach  will 
produce  results  usable  in  the  real-world  environment  as 
viewed  by  the  ultimate  decision  maker.  Analysts  and 
programmers  that  create  a  model  must  provide  documentation 
that  establishes  the  issues  examined  by  the  model, 
underlying  the  objectives  and  assumptions,  the  usability  and 
usefulness  of  the  model.  Only  with  such  documentation  can 
the  analyst  or  analysts  assisting  a  decision  maker  in  the 
resolution  of  a  particular  problem  conclude  that  the  use  of 
a  specific  model  is  appropriate.  The  real  "user"  of  a 
model  is  the  decision  maker.  The  analyst  must  recognize 
in  assisting  the  decision  maker  he  is  a  model  designer  and 
model  manipulator.  The  officer  analyst  must  gain  detailed 
knowledge   of  any  model  he  uses  in  support  of  an  analysis. 
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To   acquire   such   knowledge   a   model   must   be   adequately 
documented.      In  combat  modeling  there  are  three  levels  of 
participation  each  with  unique  documentation  requirements. 
These  levels  are: 

■  Decision-maker    level; 

■  Analyst    level; 

■  Programmer    level. 

To  enhance  the  ability  of  the  analyst  to  fully 
understand  the  model  his  documentation  should  be  presented 
from  the  context  of  discovery  rather  than  the  traditional 
context   of    justification.  This   will   allow   the   analyst      to 

know  the  alternatives  considered  and  rejected  in  structuring 
a  model  as  well  as  giving  him  greater  insights  into  the 
underlying   hypothesis   of  a    model. 

The    types  of   documentation   required  are: 

■  Decision    Maker's   Non-technical    Documentation; 

■  Analyst1 s   Conceptual-technical   Documentation; 

■  Programmer's   Technical   Documentation; 

■  Inline   Documentation. 

Since  model  modification  and  improvement  is  a  continuing 
process  the  preparation  of  formal  changes  must  be 
economically  practical  and  carefully  managed.  This  area  of 
model  documentation  is  suitable  for  a  subsequent  thesis. 
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VII.   CONCLUSIONS,  RECOMMENDATIONS,  AND  FINAL  REMARKS 


A.   SUMMAHY.  OF  PROBLEMS  DISCUSSED 


Since  1973  the  impetus  in  the  Army  modeling  community 
has  been  to  develop  scenarios  and  models  that  can  be  used 
throughout  the  Army  community  for  decisions  on  material 
reguirements  and  force  development.  The  ultimata  purpose  is 
to  bring  consistancy  into  the  decision  making 
process.  Inherent  in  this  goal  is  not  only  the  need  for 
standard  agreed  upon  scenarios  but  foe  a  repertory  of  models 
to  be  used  by  all  interested  agencies  examining  a  particular 
facet  of  the  Army.  This  ioes  not  necessarily  mean  that  only 
one  model  should  be  used  for  a  particular 
investigation.  Because  of  the  assumptions  necessary  to 
develop  any  model,  given  the  necessary  time  and  money  it  is 
best  to  exercise  more  than  one  model  in  order  to  get  an 
indication  of  how  the  assumptions  affect  the  outputs  of  each 
of  the  models.  What  is  fundamental  is  the  need  for  all 
agencies  examining  a  given  situation  to  be  able  to 
understand  each  other's  models;  this  will  provide  the  basis 
for  intelligent  discussion  of  the  pros  and  cons  of  equipment 
and  force  requirements. 

As  operations  research  and  system  analysis  have  gained 
acceptance  by  DOD  and  Department  of  the  Army,  more  combat 
models  of  various  levels  of  military  operations  have  been 
created  and  used.  Increased  use  of  models  resulted  in 
increased  levels  of  complexity.  Complexity  in  turn  caused 
new   models   to   be   created   when  potential  users  could  not 
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sufficiently  understand  and  use  existing  models  given 
available  documentation.  This  unnecessary  proliferation 
resulted  in  unexplainable  conflicting  results  which  caused 
the  credibility  of  combat  modeling  to  be  questioned.  If 
this  situation  is  to  be  resolved  unnecessary  proliferation 
must  be  checked  and  mutual  understanding  enhanced.  To 
achieve  this  goal  models  must  be  easily  transferable  between 
investigating  agencies.  Fundamental  to  this  ability  is 
complete  and  up  to  date  documentation  of  the  model  to  be 
reviewed  or  exercised.  This  documentation  will  allow  the 
potential  user  to  analyze  the  model  and  determine  its 
suitability  for  a  given  project. 

Concomitant  with  the  need  to  easily  transfer  models  is 
the  need  to  fully  understand  the  machination  by  which  a 
given  set  of  input  data  is  acted  upon  to  produce 
outputs.  If  outputs  of  a  computer  model  are  to  be  useful 
they  must  be  credible.  Conflicting  outputs  of  military 
models  are  consistently  challenged  by  some  government  agency 
of  the  DOD,  the  Congress,  the  Executive  Office  or  some  other 
branch  of  the  government.  Unless  these  challenges  are 
answered  and  conflicts  explained,  the  credibility  of  combat 
modeling  will  continue  to  suffer.  Some  criticism  is  needed 
to  purify  combat  modeling  and  identify  errors  that 
inevitably  will  appear,  some  eminates  from  those  advocating 
a  competitive  model  or  concept  and  some  of  it  is  from  those 
that  criticize  the  validity  and  usefulness  of  combat 
modeling  itself.  At  times  inadequate  documentation  has 
precluded  the  explanation  of  conflicting  results  by  the 
analyst  and  added  to  the  criticism. 

The  Comptroller  General  [15]  found  that  existing  models 
have  been  used  by  other  than  the  designer  without  thoroughly 
understanding  their  implications  and  limitations.  At  times 
this  has  resulted  in  erroneous  conclusions  being  drawn  and 
decisions   made   based  on  these  conclusions.   Subsequently, 
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the  errors  are  surfaced  with  a  loss  not  only  in  dollars 
expended  in  pursuit  of  undesirable  projects  but  in  further 
loss  of  credibility  for  theatre-level  combat  modeling. 

Adequate  documentation  will  help  alleviate  some  of  the 
adverse  publicity  and  loss  of  credibility  that  theatre-level 
combat  modeling  has  experienced  in  the  past.  To  a  great 
extent  this  has  stemmed  from  unexplainable  contradictory 
results  using  models  purporting  to  represent  the  same 
process.  To  allay  the  criticism  of  models  and  their 
outputs  and  to  enable  both  the  analyst  that  exercises  the 
model  and  the  decision  mater  that  uses  the  outputs  to  aid  in 
the  decision  making  process  the  model  must  be 
understood.  Understanding  can  be  enhanced  through  proper 
documentation.  Adequate  documentation  of  the  form  proposed 
will  facilitate  communication  between  the  decision  maker  and 
analyst  as  well  as  between  the  analyst  and  the 
programmer.  Unless  these  communication  links  are 
established  misunderstanding  of  combat  models  will  persist 
and  the  credibility  of  combat  models  and  modeling  will 
suffer  accordingly. 


B.   CONCLUSIONS  AND  RECOMMENDATIONS 


The  conclusions  drawn  from  this  research  are: 

■  The  decision  maker  is  the  "user"  of  a  model. 

■  To  be  of  value  a  model  must  be  accepted  by   the 
decision  maker. 

■  The  analyst  must  relate  the  abstractions   of  the 
model  to  the  actual  combat  process. 

■  The  analyst  is  a  model  manipulator. 

■  The   analyst   must   understand   and  explain  the 
methodology  and  structure  of  a  model. 

■  Understanding  is  deterred  by  complexity. 
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■  Models  are  becoming  more  complex. 

■  Increased   levels   of   complexity  result   in 
diminished  transparency. 

■  Inability   to   understand  and  use  existing  model 
causes  development  of  redundant  models. 

■  Unnecessary   proliferation   causes   conflicting 
results  to  be  produced  by  similar  models. 

■  Inability   to  explain  conflicting  results  is  the 
basic  cause  of  a  lack  of  credibility. 

Proliferation  can  be  reduced  through  model 
sharing.   Sharing  a  model  requires: 

■  ability  to  use  it  with  minimal  conversion; 

■  adequate  facilities  to  run  the  model; 

■  sufficient  competent  analysts  and  programmers; 

■  adequate  documentaton; 

■  formalized   irrangements   for   cost   sharing  and 
coordination. 

Prior  to  transferring  a  model  it  must  be  analyzed  for 
applicability  and  appropriateness.  Such  an  analysis  should 
consider : 

■  inputs  and  outputs; 

■  basic  causal  relationships; 
•  model  logic  and  structure; 

■  available  data  and  required  data; 

■  time  and  resources  required. 

Many  existing  combat  models  are  characterized  by: 

■  submodel  inadequacy; 

■  key  variables  excluded; 

■  inherent  inaccuracies; 

■  computational  difficulties; 

■  inconsistent  assumptions; 

■  inappropriate  assumptions; 

■  inadequate  output. 
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The  degree  to  which  meaningful  model  evaluation  can  be 
accomplished  is  significantly  influenced  by  available 
documentation.  Documentation  is  key  to  the  understanding 
of  complex  models.   The  research  conducted  indicated: 

■  Adequate  documentation  is  a  necessary  if  a 
model  is  to  be  used  by  other  than  the 
originator . 

■  Organization  exigencies  deter  adequate 
documentation. 

■  Documentation  of  combat  models  is  generally 
inadequate. 

■  If  a  model  is  poorly  documented  it  may  be  more 
economical  to  build  a  new  one  than  share  an 
existing  modal. 

■  Efforts  are  being  made  to  improve  documentation. 

The  author's  experience  with  and  research  of  combat 
model  documentation  indicates  that  there  are  three  levels  of 
interaction  with  combat  models.  These  levels  have  unique 
and  common  requirements  for  documentation.  To  satisfy  these 
requirements  the  author  envisions  three  levels  of 
documentation: 

■  DECISION    MAKER'S    NON-TECHNICAL 

■  ANALYST'S    CONCEPTUAL -TECHNICAL 

■  PROGRAMMER'S    TECHNICAL 

The  decision  maker's  and  the  programmer's  documentation 
must  provide  the  information  listed  in  chapter  six.  It  can 
be  presented  in  the  traditional  manner  using  techniques 
contained  in  most  computer  system  management  texts. 
However,  to  function  as  the  link  between  the  decision  maker 
and  the  programmer  and  to  understand  the  nuances  of  the 
model,  the  analyst  needs  documentation  that  provides  greater 
insights  than  possible  with  the  current  available 
documentation.      These      insights      will      be      provided      if      the 
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analyses  document  at  ion  is  presented  from  the  context  of 
discovery  rather  than  the  traditional  context  of 
justification. 

Many  changes  to  a  model  occur  over  extended  periods  of 
time.  The  method  of  determining  whan  it  is  economically 
practical  to  print  formal  changes,  their  disribution, 
control,  and  management  is  critical  to  the  proposed 
hierarchy  of  documentation.  These  topics  are  appropriate 
for  future  research. 


C.   FINAL  HEriARKS 


Acceptance  or  rejection  of  an  expository  thesis  in 
matters  such  as  documentation  often  depends  on  the  skill  of 
the  pleader  and  the  mood  of  the  audience.  Staring  at  the 
same  set  of  evidence  the  parties  to  the  debate  can  come  to 
sharply  different  conclusions,  since  their  preconceived 
notions  may  lead  them  to  select  and  interpret  the  evidence 
in  different  ways.  Even  though  one  may  initially  find  it 
difficult  to  believe  that  there  are  ways  to  acquire  adequate 
documentation  not  yet  tried  by  analysts  or  advocated  by 
agencies  researching  the  problem,  the  very  complexity  and 
pervasiveness  of  the  problem  suggests  the  possibility  of 
combining  the  various  proposals  in  different  ways  so  that 
some  combination  will  produce  the  desired  goal.  The 
proposal  presented  is  but  one  possible  means  to  achieve  the 
desired  end.  Of  even  more  importance  is  the  fact  that  some 
methodology  must  be  adopted  to  correct  this  lack  of  adequate 
documentation.  With  regard  to  theatre-level  combat  models, 
the  problem  has  persistel  for  almost  twenty  years.  Not  only 
has  it  made  it  near  impossible  to  easily  transfer  models 
between  interested  agencies  but  it  has  preventai  military 
analysts   from   gaining   full   knowledge   of  the  models  they 
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use. 


The  conflicting  opinisns  and  evaluations  unmasked  during 
this  research  confirm  what  is  intuitively  obvious:  many  of 
the  historical  judgements  and  decisions  concerning 
operations  research  in  general  and  theatre- level  combat 
modeling  in  particular  are  based  on  subjective  values  as 
well  as  objective  facts.  Unlike  the  natural  scientist  or 
the  analyst  using  a  simulation,  the  researcher  examining  the 
process  through  which  combat  modeling  has  evolved,  cannot 
reproduce  the  events  and  by  experimentally  altering  the 
ingredients,  change  the  result.  The  development  of  combat 
modeling  is  well  documented;  yet  controversies  have 
developed  despite  the  voluminous  sources.  Analysts  disagree 
not  because  one  may  be  more  knowledgeable  about  the  subject 
than  another,  but  because  each  weighs  and  evaluates 
differently  those  facts  of  which  both  have  knowledge.  There 
is  little  dispute  about  the  details  of  what  has  happened  in 
the  development  of  theatre-level  combat  models  but  there  is 
intense  disagreement  on  the  significance  of  past  events  and 
how  to  proceed  in  the  future.  The  analyst  has  no  fixed 
point  from  which  to  observe  the  stream  of  events  concerning 
the  development  of  theatre-level  combat  models.  Analysts 
are  borne  along  by  the  current  and  their  interpretation  of 
what  has  occurred  is  influenced  by  their  view  of  where  the 
stream  seems  to  be  headed  and  whether  the  apparent 
destination  appears  to  be  good  or  bad  for  the  enhancement 
and  development  of  the  OR  profession. 

Although  theatre-level  combat  modeling  attempts  to  be 
scientific  in  its  methods,  it  is  rarely  so  in  its 
outputs.  Outputs  are  interwoven  with  subjective  judgements, 
either  through  their  interpretation  or  by  way  of  the  inputs 
that  were  instrumental  to  producing  the  outputs  or  in  the 
very  construction  of  the  model  itself.  The  relativity  of 
subjective    judgement,   while   discouraging,   need   not   be 
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debilitating  to  theatre-level  combat  modeling.  Because 
these  models  are  the  only  means  of  examining  force  structure 
questions  vital  to  national  security,  analysts  are 
confronted  with  the  continuing  task  of  rethinking  the  basic 
structures  of  these  models.  Past  models  can  furnish  us  a 
vast  reservoir  of  experience  in  theatre- level  combat 
modeling  which  can  be  exploited  to  further  this  aspect  of 
operations  research.  However,  this  reservoir  can  be 
effectively  used  for  the  enhancement  of  the  profession  only 
if  what  has  been  accomplished  is  adequately  documented  so 
that  others  may  correctly  use  models  previously 
developed.  In  this  manner,  even  though  no  model  can  fully 
treat  all  the  intricacies  of  the  combat  process,  the  analyst 
can  enrich  the  profession  through  the  continuing  effort  to 
better  model  the  process  fully  using  all  the  knowledge  that 
has  come  before. 

The  concept  of  documentation  that  this  paper  proposes 
will  not  cure  all  the  ills.  It  is  but  a  proposal  to  correct 
a  defect,  inadequate  documentation,  that  has  long  plaqued 
theatre- level  combat  modeling.  But  if  it  is  faithfully 
executed  with  the  same  energy  and  level  of  effort  that  has 
been  expended  in  decrying  the  problem  of  inadequate 
documentation,  then  there  is  hope  that  the  omission  can  be 
corrected.  It  is  imperative  that  analysts  adequately 
document  newly  designed  models  or  modifications  to  existing 
models  so  that  other  analysts  may  use  them  properly.  Before 
a  project  is  considered  complete  it  is  the  analyst's 
responsibility  to  insure  that  the  vital  step  of 
documentation  is  accomplished.  Only  in  this  manner  can 
there  be  assurance  that  the  model  designed  or  modified  can 
be  fully  understood  by  those  who  subsequently  want  to  use 
the  model.  Analysts  using  existing  models  must  expand  and 
supplement  the  current  available  documentation  of  existing 
models  in  the  active  inventory.  The  next  time  an  existing 
model   is   used  professionalism   demands    it    be   fully 
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understood;  any  new  undocumented  factors  uncovared  in  its 
examination  should  be  formally  noted  and  made  a  permanent 
part  of  the  official  documentation.  In  this  manner  past 
omissions  will  be  corrected,  the  scientific  method  will  be 
invigorated  and  the  standing  of  the  Operations  Research 
profession  enhanced.  Subsequent  results  will  then  be  more 
fully  explainable  and  posses  greater  credibility  and  even  if 
the  conclusions  cannot  be  final,  because  of  the  impalpable 
nature  of  the  subject,  the  techniques  of  theatre-level 
combat  modeling  will  be  enriched  by  the  process. 
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